Methods, Systems, And Devices For Controlling Movement Of A Robotic Surgical System

ABSTRACT

Various exemplary methods, systems, and devices for controlling movement of a robotic surgical system are provided. In general, a plurality of surgical instruments can be simultaneously in use during performance of a surgical procedure. One or more of the plurality of instruments can be coupled to a robotic surgical system, which can be configured to control movement of the one or more of the plurality of instruments.

FIELD

The present disclosure relates generally to methods, systems, anddevices for controlling movement of a robotic surgical system.

BACKGROUND

Minimally invasive surgical (MIS) instruments are often preferred overtraditional open surgical devices due to the reduced post-operativerecovery time and minimal scarring. Laparoscopic surgery is one type ofMIS procedure in which one or more small incisions are formed in theabdomen and a trocar is inserted through the incision to form a pathwaythat provides access to the abdominal cavity. The trocar is used tointroduce various instruments and tools into the abdominal cavity, aswell as to provide insufflation to elevate the abdominal wall above theorgans. The instruments and tools can be used to engage and/or treattissue in a number of ways to achieve a diagnostic or therapeuticeffect. Endoscopic surgery is another type of MIS procedure in whichelongate flexible shafts are introduced into the body through a naturalorifice.

Various robotic systems have been developed to assist in MIS procedures.Robotic systems can allow for more intuitive hand movements bymaintaining both natural eye-hand axis. Robotic systems can also allowfor more degrees of freedom in movement by including a “wrist” joint onthe instrument, creating a more natural hand-like articulation. Onedrawback with robotic systems, however, is the loss of direct humancontact with the tissue. It can be very difficult and expensive to givetrue force feedback to the surgeon. Another drawback is that roboticsystems traditionally only allow the surgeon to control movement of upto two surgical instruments, with any other surgical instruments havingto be manually controlled by other medical personnel. It can bedifficult for the surgeon and other medical personnel to communicate andsynchronize activities of the separately controlled instruments duringperformance of a surgical procedure.

Accordingly, there remains a need for improved methods, systems, anddevices for controlling movement of a robotic surgical system.

SUMMARY

In one embodiment, a surgical system is provided that includes a roboticsurgical system and a controller. The robotic surgical system caninclude a first movable arm configured to have a first surgicalinstrument removably coupled thereto, can include a second movable armconfigured to have a second surgical instrument removably coupledthereto, can have a first mode in which the robotic surgical system isconfigured to control movement of the first arm such that the first armand the first surgical instrument removably coupled thereto moverelative to the second arm and the second surgical instrument removablycoupled thereto, can have a second mode in which the robotic surgicalsystem is configured to control movement of the first arm incoordination with the second arm such that the first arm and the firstsurgical instrument removably coupled and the second arm and the secondsurgical instrument removably coupled thereto to simultaneously move incoordination. The controller can be configured to cause the coordinatedmovement.

The surgical system can have any number of variations. For example, inthe second mode, the coordinated movement can maintain a fixed spatialrelationship between the first surgical instrument and the secondsurgical instrument. For another example, the robotic surgical systemcan be configured to receive a first user selection of which one of aplurality of arms of the robotic surgical system is the first arm, andcan be configured to receive a second user selection of which one of theplurality of arms of the robotic surgical system is the second arm. Therobotic surgical system can be configured to receive a third userselection switching the first arm from the selected one of the pluralityof instruments to another one of the plurality of instruments. For yetanother example, the robotic surgical system can be configured to switchbetween the first and second modes in response to a user command toswitch modes. For still another example, in the second mode, thecoordinated movement can move at least one of the first and secondsurgical instruments from one anatomical quadrant of the patient toanother anatomical quadrant of the patient. For another example, thefirst arm can have a sensor coupled thereto. The sensor can beconfigured to detect an impending collision of the first arm withanother arm of the robotic surgical system. The robotic surgical systemcan include a notification mechanism configured to provide anotification to a user of the robotic surgical system in response to thesensor detecting the impending collision.

In another embodiment, a surgical system is provided that includes anelectromechanical arm, a camera, and a controller. The electromechanicalarm can be configured to removably couple to a surgical instrument. Theelectromechanical arm can be configured to move so as to move thesurgical instrument removably coupled thereto relative to a patient onwhich a surgical procedure is being performed. The camera can have afield of view. The controller can be configured to receive aninstruction from a user requesting movement of the surgical instrumentfrom a start location outside of the field of view, move the surgicalinstrument in accordance with the instruction, and provide a constantcue during the movement of the surgical instrument. The cue can changeduring the movement of the surgical instrument in relation to a locationof the surgical instrument relative to the field of view. The controllercan be configured to stop the cue when the surgical instrument movesinto the field of view.

The surgical system can vary in any number of ways. For example, the cuecan be at least one of a visual cue and an auditory cue. For anotherexample, the controller can be configured to repeatedly determine acurrent bearing of a longitudinal axis of the surgical instrument as thesurgical instrument is moved by the electromechanical arm, and thecontroller can be configured to change the cue during the movement ofthe surgical instrument based on an intersection of the current bearingwith the field of view. For yet another example, after the cue has beenstopped, the controller can be configured to receive a secondinstruction from the user indicating that the user desires to adjust aposition the surgical instrument, and, if the surgical instrument movesout of the field of view as a result of the adjusting, provide anotification to the user indicating that the surgical instrument hasmoved out of the field of view. For still another example, the surgicalsystem can include a display device configured to display an image ofthe field of view visualized by the camera. The controller can beconfigured to repeatedly update the image throughout the movement of thesurgical instrument.

In another embodiment, a surgical system is provided that includes afirst electromechanical arm, a second electromechanical arm, a firstsensor attached to the first arm, and a controller configured to be inelectronic communication with the first sensor. The first arm can beconfigured to have a first surgical instrument removably coupledthereto. The first arm can be configured to move so as to adjust aposition of the first surgical instrument removably coupled theretorelative to a first surgical target. The second arm can be configured tohave a second surgical instrument removably coupled thereto. The secondarm can be configured to move so as to adjust a position of the secondsurgical instrument removably coupled thereto relative to a secondsurgical target. The first sensor can be configured to detect animpending collision between the first and second arms by determiningwhen the second arm is within a threshold minimum distance of the firstarm. The controller can be configured to trigger performance of aremedial action in response to the detected impending collision.

The surgical system can vary in any number of ways. For example, thefirst sensor can be configured to detect the impending collision usingenergy emitted by the first sensor. The emitted energy can include atleast one of light, electricity, and sound, and/or a maximum range ofthe emitted energy can define the threshold minimum distance. Foranother example, the first sensor can include a mechanical extensionextending from the first arm. The mechanical extension can be configuredto detect the impending collision by coming into contact with the secondarm. A size of the mechanical extension can define the threshold minimumdistance. For yet another example, triggering performance of theremedial action can include providing a notification to a user of thedetection of the impending collision. The notification can include atleast one of an audible sound, a visual display on a display device, ahaptic signal, and a light. The controller can trigger the notificationto be provided before the detected impending collision occurs and/orafter the detected impending collision occurs. For still anotherexample, triggering performance of the remedial action can includestopping movement of the first arm. For yet another example, triggeringperformance of the remedial action can include altering apreviously-instructed movement path of the first arm to another movementpath of the first arm determined by the controller as avoiding theimpending collision. For another example, triggering performance of theremedial action can include changing a configuration of the first arm ora configuration of the second arm. For yet another example, triggeringperformance of the remedial action can include moving the second arm toavoid the impending collision. For another example, the second arm canhave a second sensor coupled thereto, the second sensor can beconfigured to detect an impending collision between the first and secondarms by determining when the first arm is within a threshold minimumdistance of the second arm, the controller can be configured to be inelectronic communication with the second sensor, and the controller canbe configured to trigger performance of a remedial action in response tothe impending collision detected by the second sensor. For yet anotherexample, the surgical system can include a user input device configuredto receive an input from a user indicating a desired movement of thefirst instrument removably coupled to the first arm. The controller canbe configured to be in electronic communication with the user inputdevice and the first arm, and the controller can be configured tocontrol movement of the first arm based on the input received by theuser input device.

In another aspect, a surgical method is provided that in one embodimentincludes receiving a user selection of one of a plurality of surgicalinstruments as a master instrument. The plurality of surgicalinstruments can be controlled by a robotic surgical system duringperformance of a surgical procedure on a patient. Each of the pluralityof surgical instruments can be coupled to one of a plurality of movablearms of the robotic surgical system. The method can include receiving auser selection of one or more other ones of the plurality of surgicalinstruments as follower instruments, receiving an input indicatingmovement of the master instrument, and causing the one or more followerinstruments to move in coordination with the movement of the masterinstrument so as to maintain a fixed spatial relationship between themaster instrument and the one or more follower instruments throughoutthe movement of the master instrument.

The surgical method can vary in any number of ways. For example, themovement of the master instrument can be from one anatomical quadrant ofthe patient to another anatomical quadrant of the patient. For anotherexample, the plurality of surgical instruments can be located indifferent anatomical quadrants of the patient. For yet another example,the surgical method can include receiving an input indicating auser-requested change in visualization provided by a visualizationdevice of at least one of the plurality of surgical instruments, andeffecting the user-requested change in the visualization provided by thevisualization device. The change can be effected simultaneously with themovement of the master instrument and the one or more followerinstruments such that the visualization device, the master instrument,and the one or more follower instruments move at the same time, or thechange can be effected either before or after the movement of the masterinstrument and the one or more follower instruments such that thevisualization device moves at a different time than the masterinstrument and the one or more follower instruments. For still anotherexample, the surgical method can include displaying on a display deviceshowing the plurality of surgical instruments an indication of which oneof the plurality of surgical instruments has been selected as the masterinstrument and an indication of which one or more of the plurality ofsurgical instruments has been selected as the one or more followerinstruments. For another example, the robotic surgical system caninclude a controller that receives the user selection of the masterinstrument, receives the user selection of the one or more followerinstruments, receives the input indicating the movement of the masterinstrument, and causes the one or more follower instruments to move incoordination with the movement of the master instrument. For yet anotherexample, the robotic surgical system can include a first user inputdevice and a controller in electronic communication with one another,and receiving the input indicating movement of the master instrument caninclude receiving at the controller an input from the first user inputdevice. The robotic surgical system can include a second user inputdevice in electronic communication with the controller. The second userinput device can be associated with a visualization device providingvisualization of at least one of the plurality of surgical instruments.The method can include receiving an input at the controller from thesecond user input device indicating a user-requested change in thevisualization provided by the visualization device, and effecting theuser-requested change in the visualization provided by the visualizationdevice.

In another embodiment, a surgical method is provided that includesdisplaying an image on a display device showing a field of view of acamera visualizing a surgical area within a patient, and providing aconstant cue to a user positioning a distal working end of a surgicalinstrument relative to the patient using a robotic surgical system towhich the surgical instrument is removably coupled. The distal workingend of a surgical instrument can start its movement from outside thefield of view. The cue can change over time in relation to a location ofthe distal working end of the surgical instrument relative to the fieldof view displayed on the display device. The surgical method can includestopping provision of the cue in response to the distal working end ofthe surgical instrument entering the field of view displayed on thedisplay device.

The surgical method can have any number of variations. For example, thecue can be at least one of a visual cue and an auditory cue. For anotherexample, the cue can include a light that changes over time in at leastone of brightness and color in relation to the location of the distalworking end of the surgical instrument relative to the field of viewdisplayed on the display device. For yet another example, the cue caninclude an icon shown on the display device that changes over time inrelation to the location of the distal working end of the surgicalinstrument relative to the field of view displayed on the displaydevice. For still another example, the cue can include an audible soundthat changes over time in relation to the location of the distal workingend of the surgical instrument relative to the field of view displayedon the display device. The audible sound changing can include changing apitch of the sound or changing an amplitude of the sound. For anotherexample, the cue changing over time in relation to the location of thedistal working end of the surgical instrument relative to the field ofview displayed on the display device can be based on an intersection ofthe field of view and a trajectory defined by a longitudinal axis of anelongate shaft of the surgical instrument. For still another example,the cue changing over time in relation to the location of the distalworking end of the surgical instrument relative to the field of viewdisplayed on the display device can be based on an intersection of thefield of view and a trajectory defined by a longitudinal axis of atrocar through which the surgical instrument is advanced into thepatient. For another example, the cue can change over time based on atleast one of a direction that the distal working end of the surgicalinstrument is moving in relative to the field of view displayed on thedisplay device, a distance between the distal working end of thesurgical instrument and the field of view displayed on the displaydevice, and whether the distal working end of the surgical instrument ismoving closer to or farther from the field of view displayed on thedisplay device. For yet another example, the surgical method caninclude, after the cue has stopped being provided, adjusting a positionof the distal working end of the surgical instrument, and, if the distalworking end of the surgical instrument moves out of the field of view asa result of the adjusting, providing a notification to the userindicating that the distal working end of the surgical instrument hasmoved out of the field of view. For still another example, the roboticsurgical system can include a plurality of electromechanical arms. Thesurgical instrument can be removably coupled to one of the arms. Thelocation of the distal working end of the surgical instrument relativeto the field of view displayed on the display device can be determinedbased on a position of the one of the arms to which the surgicalinstrument is removably coupled.

In another embodiment, a surgical method is provided that includesreceiving an input from a user indicating that the user desires to movea surgical instrument currently out of a field of view of a cameravisualizing a surgical area of a patient. The surgical instrument can becoupled to an electromechanical arm of a robotic surgical system thatmoves the surgical instrument relative to the patient. The surgicalmethod can include calculating a bearing of a trajectory line between atrocar, through which the surgical instrument is advanced, and a pointin the surgical area being visualized in the field of view. The surgicalmethod can include repeatedly determining a current bearing of alongitudinal axis of the surgical instrument as the surgical instrumentis moved by movement of the electromechanical arm, and providing a cueindicative of a difference between the bearing of the trajectory lineand the current bearing of the longitudinal axis. The cue can changeover time based on an amount of the difference as the surgicalinstrument moves. The surgical method can include stopping provision ofthe cue when the difference becomes substantially zero.

The surgical method can have any number of variations. For example,providing the cue can include at least one of providing a visual cue andproviding an auditory cue. For another example, the surgical method caninclude displaying an image on a display device showing the field ofview of the camera. For yet another example, the surgical method caninclude, after the cue has stopped being provided, receiving a secondinput from the user indicating that the user desires to adjust aposition the surgical instrument currently in the field of view of thecamera, and, if the surgical instrument moves out of the field of viewas a result of the adjusting, providing a notification to the userindicating that the surgical instrument has moved out of the field ofview.

In another embodiment, a surgical method is provided that includesmoving a first electromechanical arm of a robotic surgical system thatincludes one or more additional electromechanical arms. Each of the armscan have a surgical instrument coupled thereto. The surgical method caninclude sensing an impending collision between the moving first arm andone of the one or more additional electromechanical arms using a sensorattached to the first arm, and in response to sensing the impendingcollision, performing a remedial action that addresses the sensedimpending collision prior to occurrence of the sensed impendingcollision.

The method can vary in any number of ways. For example, performing theremedial action can include providing a notification to a user of thesensed impending collision. The notification includes at least one of anaudible sound, a visual display on a display device, a haptic signal,and a light. For another example, performing the remedial action caninclude stopping the movement of the first arm. For yet another example,the movement of the first arm can be along a predetermined path definedby a user, and performing the remedial action can include changing thepredetermined path to avoid the impending collision. For still anotherexample, performing the remedial action can include changing a shape ofthe first arm or a shape of the one of the one or more additionalelectromechanical arms.

In another embodiment, a surgical method is provided that includesdisplaying on a display device an image of a surgical area visualized bya camera. The camera has a field of view defining a perimeter of thevisualized surgical area. The surgical method can include providing asignal indicative of a current position of a surgical instrument, whichcan be located outside the perimeter of the visualized surgical area,relative to the visualized surgical area. The surgical instrument can becoupled to an electromechanical arm of a robotic surgical system. Thesurgical method can include changing the signal in real time withmovement of the electromechanical arm and the surgical instrumentcoupled thereto relative to the visualized surgical area based on thecurrent position of the surgical instrument relative to the surgicalarea.

The surgical method can vary in any number of ways. For example, thesurgical method can include stopping the signal in response to thesurgical instrument moving to a location inside the perimeter of thevisualized surgical area. For another example, the surgical method caninclude determining the current position of the surgical instrumentrelative to the surgical area based on a position of a longitudinal axisof the surgical instrument relative to the visualized surgical area. Thesignal can be changed in real time based on an intersection of thelongitudinal axis with the visualized surgical area. For yet anotherexample, the surgical method can include determining the currentposition of the surgical instrument relative to the surgical area basedon a position of a longitudinal axis of a trocar, through which thesurgical instrument is advanced, with the visualized surgical area. Thesignal can be changed in real time based on an intersection of thelongitudinal axis with the visualized surgical area. For still anotherexample, the signal can be changed in real time based on a proximity ofa current position of the surgical instrument to a reference pointwithin of the visualized surgical area. For another example, the signalcan be changed in real time based on a proximity of a current positionof a trocar, through which the surgical instrument is advanced, to areference point within the visualized surgical area. For still anotherexample, the surgical method can include calculating the currentposition of the surgical instrument relative to the surgical area usinga controller of the robotic surgical system. For yet another example,the signal can include at least one of an audible sound, a visual signalshown on the display device, a visual signal shown on a second displaydevice, a tactile signal palpable by a user of the robotic surgicalsystem, and an illuminated light. The signal can include at least theaudible sound, and changing the signal can include changing a pitch ofthe sound or changing an amplitude of the sound. The signal can includeat least one of the visual signal shown on the display device and thevisual signal shown on the second display device, and changing thesignal can include changing an appearance of the at least one of thevisual signal shown on the display device and the visual signal shown onthe second display device. The signal can include at least theilluminated light, and changing the signal can include changing thelight in at least one of brightness and color.

In another embodiment, a surgical method is provided that includesreceiving a user input at a robotic surgical system. The user input canrequest movement of a surgical instrument removably and replaceablycoupled to a first electrosurgical arm of the robotic surgical system.The surgical method can include causing movement of the firstelectrosurgical arm in response to the received user input, therebycausing movement of the surgical instrument removably and replaceablycoupled to the first electrosurgical arm. The surgical method caninclude determining in real time with the movement of the firstelectrosurgical arm whether the first electrosurgical arm is within athreshold minimum distance of another portion of the robotic surgicalsystem, and, in response to determining that the first electrosurgicalarm is within the threshold minimum distance, triggering performance ofa remedial action to reduce a chance of a collision between the movingfirst electrosurgical arm and the other portion of the robotic surgicalsystem.

The surgical method can have any number of variations. For example, theperformance of the remedial action can include at least one of slowing aspeed of the moving first electrosurgical arm, adjusting the movement ofthe first electrosurgical arm in contradiction to the movement requestedby the user input, moving the other portion of the robotic surgicalsystem, and providing a notification to a user indicating that the firstelectrosurgical arm is within the threshold minimum distance. Foranother example, the other portion of the robotic surgical system caninclude a second electromechanical arm of the robotic surgical systemthat is configured to removably and replaceably couple to a secondsurgical instrument.

In another embodiment, a surgical method is provided that includesreceiving a first user input at a robotic surgical system. The firstuser input can request movement of a first surgical instrument coupledto a first electromechanical arm of the robotic surgical system. Thesurgical method can include, in response to the received first userinput, moving the first electromechanical arm in a first mode of therobotic surgical system in which the first electromechanical arm movesrelative to a second electromechanical arm of the robotic surgicalsystem such that a relative position of the first and secondelectromechanical arms changes. The surgical method can includereceiving a second user input at the robotic surgical system. The seconduser input can request another movement of the first surgicalinstrument. The surgical method can include, in response to the receivedsecond user input, moving the first electromechanical arm in a secondmode of the robotic surgical system in which the other movement of thefirst electromechanical arm causes corresponding movement of the secondelectromechanical arm such that the relative position of the first andsecond electromechanical arms is maintained.

The surgical method can vary in any number of ways. For example, thesurgical method can include receiving a third user input at the roboticsurgical system. The third user input can set the robotic surgicalsystem in one of the first and second modes. For another example, therobotic surgical system can include one or more additionalelectromechanical arms. In the first mode, the first electromechanicalarm can move relative to each of the one or more additionalelectromechanical arms. In the second mode, the other movement of thefirst electromechanical arm can cause corresponding movement of each ofthe one or more additional electromechanical arms. For yet anotherexample, the robotic surgical system can include a plurality ofelectromechanical arms in addition to the first electromechanical arm.The surgical method can include receiving a third user input at therobotic surgical system indicating a user choice of which of theplurality of electromechanical arms is the second electromechanical armthat moves with the first electromechanical arm in the second mode.

Non-transitory computer program products (i.e., physically embodiedcomputer program products) are also provided that store instructions,which when executed by one or more processors of one or more computersystems, causes at least one processor to perform operations herein.Similarly, computer systems are also provided that can include one ormore processors and one or more memories coupled to the one or moreprocessors. Each of the one or more memories can temporarily orpermanently store instructions that cause at least one processor toperform one or more of the operations described herein. In addition,methods can be implemented by one or more processors either within asingle computer system or distributed among two or more computersystems. Such computer systems can be connected and can exchange dataand/or commands or other instructions or the like via one or moreconnections, including but not limited to a connection over a network(e.g., the Internet, a wireless wide area network, a local area network,a wide area network, a wired network, etc.), via a direct connectionbetween one or more of the multiple computer systems, etc.

BRIEF DESCRIPTION OF DRAWINGS

This invention will be more fully understood from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a graphical representation of terminology associated with sixdegrees of freedom;

FIG. 2 is a schematic view of one embodiment of a computer system;

FIG. 3 is a schematic view of one embodiment of a robotic surgicalsystem configured to be operated by a user and to be used duringperformance of a surgical procedure on a patient;

FIG. 4 is a perspective view of one embodiment of an arm of a roboticsurgical system, the arm being mounted to a surgical table;

FIG. 5 is a perspective view of an active portion of the arm of FIG. 4;

FIG. 6 is a perspective view of one embodiment of a robotic surgicalsystem;

FIG. 7 is a schematic view of one embodiment of the robotic surgicalsystem of FIG. 6 in use during performance of a surgical procedure on apatient;

FIG. 8 is a perspective view of the robotic surgical system of FIG. 7 inuse during performance of the surgical procedure on a patient;

FIG. 9 is a schematic and perspective view of another embodiment of arobotic surgical system;

FIG. 10 is a perspective view of one embodiment of a master tool in afield generated by a transmitter of the robotic surgical system of FIG.9;

FIG. 11 is a schematic view of another embodiment of a robotic surgicalsystem having a plurality of surgical instruments coupled thereto;

FIG. 12 is a schematic view of a field of view of a camera;

FIG. 13 is a flowchart of one embodiment of a process of providing acue;

FIG. 14 is a flowchart of another embodiment of a process of providing acue;

FIG. 15 is a schematic view of one embodiment of use of the roboticsurgical system and the plurality of surgical instruments of FIG. 11;

FIG. 16 is a schematic view of the use of the robotic surgical systemand the plurality of surgical instruments of FIG. 15 at a timesubsequent to a time of FIG. 15;

FIG. 17 is a schematic view of the use of the robotic surgical systemand the plurality of surgical instruments of FIG. 16 at a timesubsequent to a time of FIG. 16;

FIG. 18 is a schematic view of another embodiment of a robotic surgicalsystem;

FIG. 19 is a flowchart of one embodiment of a process of detecting apossible collision in a robotic surgical system using one or moresensors;

FIG. 20 is a flowchart of another embodiment of a process of detecting apossible collision in a robotic surgical system using one or moresensors;

FIG. 21 is a flowchart of one embodiment of a process of reacting to adetection of a possible collision in a robotic surgical system;

FIG. 22 is a schematic view of another embodiment of a robotic surgicalsystem;

FIG. 23 is a flowchart showing one embodiment of a first portion of aprocess of a robotic surgical system facilitating movement of a surgicalinstrument between different anatomical quadrants;

FIG. 24 is a flowchart showing a second portion of the process of FIG.23;

FIG. 25 is a schematic view of one embodiment of a display showing avisualized surgical space;

FIG. 26 is a schematic view of the display of FIG. 25 at a subsequentpoint in time;

FIG. 27 is a schematic view of another embodiment of a display showing avisualized surgical space;

FIG. 28 is a schematic view of the display of FIG. 27 at a subsequentpoint in time;

FIG. 29 is a schematic view of the display of FIG. 28 at a subsequentpoint in time;

FIG. 30 is a schematic view of the display of FIG. 29 at a subsequentpoint in time;

FIG. 31 is a schematic view of the display of FIG. 30 at a subsequentpoint in time;

FIG. 32 is a schematic view of the display of FIG. 31 at a subsequentpoint in time;

FIG. 33 is a schematic view of another embodiment of a display showing avisualized surgical space;

FIG. 34 is a schematic view of the display of FIG. 33 at a subsequentpoint in time;

FIG. 35 is a schematic view of the display of FIG. 34 at a subsequentpoint in time;

FIG. 36 is a schematic view of the display of FIG. 35 at a subsequentpoint in time.

DETAILED DESCRIPTION

Certain exemplary embodiments will now be described to provide anoverall understanding of the principles of the structure, function,manufacture, and use of the devices and methods disclosed herein. One ormore examples of these embodiments are illustrated in the accompanyingdrawings. Those skilled in the art will understand that the devices andmethods specifically described herein and illustrated in theaccompanying drawings are non-limiting exemplary embodiments and thatthe scope of the present invention is defined solely by the claims. Thefeatures illustrated or described in connection with one exemplaryembodiment may be combined with the features of other embodiments. Suchmodifications and variations are intended to be included within thescope of the present invention.

Further, in the present disclosure, like-named components of theembodiments generally have similar features, and thus within aparticular embodiment each feature of each like-named component is notnecessarily fully elaborated upon. Additionally, to the extent thatlinear or circular dimensions are used in the description of thedisclosed systems, devices, and methods, such dimensions are notintended to limit the types of shapes that can be used in conjunctionwith such systems, devices, and methods. A person skilled in the artwill recognize that an equivalent to such linear and circular dimensionscan easily be determined for any geometric shape. Sizes and shapes ofthe systems and devices, and the components thereof, can depend at leaston the anatomy of the subject in which the systems and devices will beused, the size and shape of components with which the systems anddevices will be used, and the methods and procedures in which thesystems and devices will be used.

Various exemplary methods, systems, and devices for controlling movementof a robotic surgical system are provided. In general, a plurality ofsurgical instruments can be simultaneously in use during performance ofa surgical procedure. One or more of the plurality of instruments can becoupled to a robotic surgical system, which can be configured to controlmovement of the one or more of the plurality of instruments. In someembodiments, the robotic surgical system can be configured to controlmovement of all of the plurality of instruments. In other embodiments,the robotic surgical system can be configured to control movement of atleast one of the plurality of instruments, and movement of a remainderof the plurality of surgical instruments can be controlled in anotherway, e.g., by another robotic surgical system or through manual usercontrol.

During performance of the surgical procedure, movement of the pluralityof surgical instruments by the robotic surgical system relative to oneor more others of the surgical instruments and/or relative to a surgicalsite can create any one or more of a variety of challenges. For example,it can be difficult for the robotic surgical system to move one of thesurgical instruments from a position out of frame of vision of anotherone of the surgical instruments configured to provide visualization to aposition within the frame of vision because, e.g., the frame of visioncan be fixed to a specific surgical area of interest and not to the areafrom which the one of the surgical instruments is being moved, such asfrom outside the patient's body. For another example, movement of one ormore of the surgical instruments by the robotic surgical system can riskcollision between objects, such as between a part of the roboticsurgical system (e.g., a part moving to move the surgical instrument)and a non-moving one of the surgical instruments, another part of therobotic surgical system, a mounted light illuminating the surgical area,a surgical table, etc., which can risk damage to one or both of thecolliding objects and/or can risk harming the patient. For yet anotherexample, controlling movement of any one or more of the surgicalinstruments from one location to another using the robotic surgicalsystem can be difficult when the movement is from one anatomicalquadrant of a patient to another, e.g., from one quadrant of thepatient's abdomen to another quadrant of the abdomen, because in atleast some robotic surgical systems each quadrant may have its ownreference Cartesian frame. For still another example, when one of thesurgical instruments moves, it may move out of a frame of vision ofanother one of the surgical instruments configured to providevisualization (e.g., a camera), which can make the location of the oneof the surgical instruments and/or the surgical area more difficult fora surgeon and/or other medical personnel to assess.

At least some of the methods, systems, and devices for controllingmovement of a robotic surgical system provided herein can be configuredto facilitate movement of a surgical instrument into a field of view ofa camera visualizing a surgical area. In general, a robotic surgicalsystem having a surgical instrument coupled thereto can be configured tosignal a user of the robotic surgical system with information regardinga location of the surgical instrument relative to a field of view of acamera visualizing a surgical area, e.g., relative to an area of spacecurrently being visualized by a lens of the camera. The information canallow the user to control movement of the surgical instrument by therobotic surgical system to effectively guide the surgical instrumentfrom out of the camera's field of view and into the camera's field ofview. The signal can be configured to change in real time with thesurgical instrument's movement based on a position of the surgicalinstrument relative to the field of view and/or based on a position ofan introducer device (e.g., a cannula, a trocar, etc.) through which thesurgical instrument is advanced to gain access to a patient's body. Thesignal may thus facilitate accurate guidance of the surgical instrumentto the field of view, may allow for corrective movement of the surgicalinstrument based on the signal's change (e.g., correct for theinstrument being moved off course from intersection with the field ofview, etc.), and/or may facilitate “blind” positioning of the surgicalinstrument by allowing the surgical instrument to be accurately guidedto the field of view without the surgical instrument being visualized bythe user (direct visualization or indirect visualization on a displaydevice) during the instrument's movement to the field of view. Thesignal can be configured to be visualized by the user (e.g., a light,text on a display device, an icon on a display device, etc.), to beheard by the user (e.g., an audible tone, a computerized voice, etc.),or to be tactilely felt by the user (e.g., a vibration, heat, cooling,etc.). The signal include one or more signals, and the signal can bevisual, auditory, and/or tactile.

At least some of the methods, systems, and devices for controllingmovement of a robotic surgical system provided herein can be configuredto facilitate detection of collision of a portion of the roboticsurgical system with another object. In general, a robotic surgicalsystem can be configured to detect a collision between two objectsbefore the collision happens. Detecting the collision before it occursmay allow the robotic surgical system to trigger performance of acorrective action aimed at preventing the collision from occurring,thereby reducing chances of the collision occurring and, in the eventthat the collision nevertheless occurs, reduce chances of the collidingobjects from being damaged by reducing severity of the collision (e.g.,by reducing an impact force between the colliding objects, byreorienting the position of the collided portion of the robotic surgicalsystem prior to the collision, etc.). In an exemplary embodiment, therobotic surgical system can include a movable arm configured to have asurgical instrument coupled thereto, and the robotic surgical system canbe configured to, while the arm is moving with the surgical instrumentcoupled thereto, detect a collision of the arm with another objectbefore the collision occurs and trigger performance of a correctiveaction to help prevent the collision from occurring.

At least some of the methods, systems, and devices for controllingmovement of a robotic surgical system provided herein can be configuredto facilitate movement of a surgical instrument between differentanatomical quadrants. In general, a robotic surgical system having asurgical instrument coupled thereto can be configured to controlmovement of a master surgical instrument coupled to the robotic surgicalsystem in response to user input to the robotic surgical system, and tocause corresponding movement of one or more follower surgicalinstruments coupled to the robotic surgical system. The robotic surgicalsystem may thus facilitate movement of surgical instruments betweendifferent anatomical quadrants by allowing instruments to move incorrespondence to another surgical instrument and/or may reduce chancesof instrument collision since multiple instruments can correspondinglymove so as to maintain their spatial relationship. The robotic surgicalsystem can be configured to allow the user to select the followersurgical instruments from among all surgical instruments coupled to therobotic surgical system, which may allow the user to select coordinatedinstrument movement on an as-needed basis during performance of asurgical procedure, may allow different instruments during the course ofa surgical procedure to serve as a master surgical instrument that otherinstrument(s) follow, and/or may allow zero follower instruments to beselected such that the master surgical instrument can move relative toall the other surgical instruments coupled to the robotic surgicalsystem. In an exemplary embodiment, first and second surgicalinstruments can be coupled to the robotic surgical system, with thefirst surgical instrument including a camera. The robotic surgicalsystem can thus be configured to allow the camera to follow movement ofthe second surgical instrument, e.g., the camera is selected as thefollower instrument and the second surgical instrument is selected asthe master instrument, which can help maintain visualization of thesecond surgical instrument during movement thereof. The robotic surgicalsystem can also thus be configured to allow the second surgicalinstrument to follow movement of the camera, e.g., the second surgicalinstrument is selected as the follower instrument and the camera isselected as the master instrument, which can allow the second surgicalinstrument, when visualized by the camera, to remain within the camera'svision during the camera's movement.

TERMINOLOGY

There are a number of ways in which to describe the movement ofcomponents of a surgical system, as well as its position and orientationin space. One particularly convenient convention is to characterizemovement in terms of degrees of freedom. The degrees of freedom are thenumber of independent variables that uniquely identify a component'spose (e.g., location defined by translation and orientation defined byrotation) or configuration. The set of Cartesian degrees of freedom isusually represented by the three translational or position variables,e.g., surge, heave, sway, and by the three rotational or orientationvariables, e.g., Euler angles or roll, pitch, yaw, that describe theposition and orientation of a component of a surgical system withrespect to a given reference Cartesian frame. As used herein, and asillustrated in FIG. 1, the term “surge” refers to forward and backwardmovement, the term “heave” refers to movement up and down, and the term“sway” refers to movement left and right. With regard to the rotationalterms, “roll” refers to tilting side to side, “pitch” refers to tiltingforward and backward, and “yaw” refers to turning left and right. In amore general sense, each of the translation terms refers to movementalong one of the three axes in a Cartesian frame, and each of therotational terms refers to rotation about one of the three axes in aCartesian frame.

Although the number of degrees of freedom is at most six, a condition inwhich all the translational and orientational variables areindependently controlled, the number of joint degrees of freedom isgenerally the result of design choices that involve considerations ofthe complexity of the mechanism and the task specifications. Fornon-redundant kinematic chains, the number of independently controlledjoints is equal to the degree of mobility for an end effector. Forredundant kinematic chains, the end effector will have an equal numberof degrees of freedom in Cartesian space that will correspond to acombination of translational and rotational motions. Accordingly, thenumber of degrees of freedom can be more than, equal to, or less thansix.

With regard to characterizing the position of various components of thesurgical system and the mechanical frame, the terms “forward” and“rearward” may be used. In general, the term “forward” refers to an endof the surgical system that is closest to the distal end of the inputtool, and when in use in a surgical procedure, to the end disposedwithin a patient's body. The term “rearward” refers to an end of thesurgical system farthest from the distal end of the input tool, and whenin use, generally to the end farther from the patient.

The terminology used herein is not intended to limit the invention. Forexample, spatially relative terms, e.g., “superior,” “inferior,”“beneath,” “below,” “lower,” “above,” “upper,” “rearward,” “forward,”etc., may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positionsand orientations of the device in use or operation in addition to theposition and orientation shown in the figures. For example, if thedevice in the figures is turned over, elements described as “inferiorto” or “below” other elements or features would then be “superior to” or“above” the other elements or features. Likewise, descriptions ofmovement along and around various axes includes various special devicepositions and orientations. As will be appreciated by those skilled inthe art, specification of the presence of stated features, steps,operations, elements, and/or components does not preclude the presenceor addition of one or more other features, steps, operations, elements,components, and/or groups described herein. In addition, componentsdescribed as coupled may be directly coupled, or they may be indirectlycoupled via one or more intermediate components.

There are several general aspects that apply to the various descriptionsbelow. For example, at least one surgical end effector is shown anddescribed in various figures. An end effector is the part of a minimallyinvasive or invasive surgical instrument or assembly that performs aspecific surgical function, e.g., forceps/graspers, needle drivers,scissors, electrocautery hooks, staplers, clip appliers/removers,suction tools, irrigation tools, etc. Any end effector can be utilizedwith the surgical system described herein. Further, in exemplaryembodiments, an end effector can be configured to be manipulated by auser input tool. The input tool can be any tool that allows successfulmanipulation of the end effector, whether it be a tool similar in shapeand style to the end effector, such as an input tool of scissors similarto end effector scissors, or a tool that is different in shape and styleto the end effector, such as an input tool of a glove dissimilar to endeffector graspers, and such as input tool of a joystick dissimilar toend effector graspers. In at least some embodiments, the input tool canbe a larger scaled version of the end effector to facilitate ease ofuse. Such a larger scale input tool can have finger loops or grips of asize suitable for a user to hold. However, the end effector and theinput tool can have any relative size.

A slave tool, e.g., a surgical instrument, of the surgical system can bepositioned inside a patient's body cavity through an access point in atissue surface for minimally invasive surgical procedures. Typically,cannulas such as trocars are used to provide a pathway through a tissuesurface and/or to prevent a surgical instrument or guide tube fromrubbing on patient tissue. Cannulas can be used for both incisions andnatural orifices. Some surgical procedures require insufflation, and thecannula can include one or more seals to prevent excess insufflation gasleakage past the instrument or guide tube. In at least some embodiments,the cannula can have a housing coupled thereto with two or more sealedports for receiving various types of instruments besides the slaveassembly. As will be appreciated by a person skilled in the art, any ofthe surgical system components disclosed herein can have a functionalseal disposed thereon, therein, and/or therearound to prevent and/orreduce insufflation leakage while any portion of the surgical system isdisposed through a surgical access port, such as a cannula. The surgicalsystem can also be used in open surgical procedures. As used herein, asurgical access point is a point at which the slave tool enters a bodycavity through a tissue surface, whether through a cannula in aminimally invasive procedure or through an incision in an openprocedure.

Computer Systems

The systems, devices, and methods disclosed herein can be implementedusing one or more computer systems, which may also be referred to hereinas digital data processing systems and programmable systems.

One or more aspects or features of the subject matter described hereincan be realized in digital electronic circuitry, integrated circuitry,specially designed application specific integrated circuits (ASICs),field programmable gate arrays (FPGAs) computer hardware, firmware,software, and/or combinations thereof. These various aspects or featurescan include implementation in one or more computer programs that areexecutable and/or interpretable on a programmable system including atleast one programmable processor, which can be special or generalpurpose, coupled to receive data and instructions from, and to transmitdata and instructions to, a storage system, at least one input device,and at least one output device. The programmable system or computersystem may include clients and servers. A client and server aregenerally remote from each other and typically interact through acommunication network. The relationship of client and server arises byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other.

The computer programs, which can also be referred to as programs,software, software applications, applications, components, or code,include machine instructions for a programmable processor, and can beimplemented in a high-level procedural language, an object-orientedprogramming language, a functional programming language, a logicalprogramming language, and/or in assembly/machine language. As usedherein, the term “machine-readable medium” refers to any computerprogram product, apparatus and/or device, such as for example magneticdiscs, optical disks, memory, and Programmable Logic Devices (PLDs),used to provide machine instructions and/or data to a programmableprocessor, including a machine-readable medium that receives machineinstructions as a machine-readable signal. The term “machine-readablesignal” refers to any signal used to provide machine instructions and/ordata to a programmable processor. The machine-readable medium can storesuch machine instructions non-transitorily, such as for example as woulda non-transient solid-state memory or a magnetic hard drive or anyequivalent storage medium. The machine-readable medium can alternativelyor additionally store such machine instructions in a transient manner,such as for example as would a processor cache or other random accessmemory associated with one or more physical processor cores.

To provide for interaction with a user, one or more aspects or featuresof the subject matter described herein can be implemented on a computerhaving a display device, such as for example a cathode ray tube (CRT) ora liquid crystal display (LCD) or a light emitting diode (LED) monitorfor displaying information to the user and a keyboard and an inputdevice, e.g., a mouse, a trackball, a hand tracker, a gesturerecognition device, Kinect, etc., by which the user may provide input tothe computer. Other kinds of devices can be used to provide forinteraction with a user as well. For example, feedback provided to theuser can be any form of sensory feedback, such as for example visualfeedback, auditory feedback, or tactile feedback; and input from theuser may be received in any form, including, but not limited to,acoustic, speech, or tactile input. Other possible input devicesinclude, but are not limited to, touch screens or other touch-sensitivedevices such as single or multi-point resistive or capacitive trackpads,voice recognition hardware and software, optical scanners, opticalpointers, digital image capture devices and associated interpretationsoftware, and the like.

FIG. 2 illustrates one exemplary embodiment of a computer system 100. Asshown, the computer system 100 can include one or more processors 102which can control the operation of the computer system 100. “Processors”are also referred to herein as “controllers.” The processor(s) 102 caninclude any type of microprocessor or central processing unit (CPU),including programmable general-purpose or special-purposemicroprocessors and/or any one of a variety of proprietary orcommercially available single or multi-processor systems. The computersystem 100 can also include one or more memories 104, which can providetemporary storage for code to be executed by the processor(s) 102 or fordata acquired from one or more users, storage devices, and/or databases.The memory 104 can include read-only memory (ROM), flash memory, one ormore varieties of random access memory (RAM) (e.g., static RAM (SRAM),dynamic RAM (DRAM), or synchronous DRAM (SDRAM)), and/or a combinationof memory technologies.

The various elements of the computer system 100 can be coupled to a bussystem 112. The illustrated bus system 112 is an abstraction thatrepresents any one or more separate physical busses, communicationlines/interfaces, and/or multi-drop or point-to-point connections,connected by appropriate bridges, adapters, and/or controllers. Thecomputer system 100 can also include one or more network interface(s)106, one or more input/output (TO) interface(s) 108, and one or morestorage device(s) 110.

The network interface(s) 106 can enable the computer system 100 tocommunicate with remote devices, e.g., other computer systems, over anetwork, and can be, for non-limiting example, remote desktop connectioninterfaces, Ethernet adapters, and/or other local area network (LAN)adapters. The IO interface(s) 108 can include one or more interfacecomponents to connect the computer system 100 with other electronicequipment. For non-limiting example, the IO interface(s) 108 can includehigh speed data ports, such as universal serial bus (USB) ports, 1394ports, Wi-Fi, Bluetooth, etc. Additionally, the computer system 100 canbe accessible to a human user, and thus the IO interface(s) 108 caninclude displays, speakers, keyboards, pointing devices, and/or variousother video, audio, or alphanumeric interfaces. The storage device(s)110 can include any conventional medium for storing data in anon-volatile and/or non-transient manner. The storage device(s) 110 canthus hold data and/or instructions in a persistent state, i.e., thevalue is retained despite interruption of power to the computer system100. The storage device(s) 110 can include one or more hard disk drives,flash drives, USB drives, optical drives, various media cards,diskettes, compact discs, and/or any combination thereof and can bedirectly connected to the computer system 100 or remotely connectedthereto, such as over a network. In an exemplary embodiment, the storagedevice(s) can include a tangible or non-transitory computer readablemedium configured to store data, e.g., a hard disk drive, a flash drive,a USB drive, an optical drive, a media card, a diskette, a compact disc,etc.

The elements illustrated in FIG. 2 can be some or all of the elements ofa single physical machine. In addition, not all of the illustratedelements need to be located on or in the same physical machine.Exemplary computer systems include conventional desktop computers,workstations, minicomputers, laptop computers, tablet computers,personal digital assistants (PDAs), mobile phones, and the like.

The computer system 100 can include a web browser for retrieving webpages or other markup language streams, presenting those pages and/orstreams (visually, aurally, or otherwise), executing scripts, controlsand other code on those pages/streams, accepting user input with respectto those pages/streams (e.g., for purposes of completing input fields),issuing HyperText Transfer Protocol (HTTP) requests with respect tothose pages/streams or otherwise (e.g., for submitting to a serverinformation from the completed input fields), and so forth. The webpages or other markup language can be in HyperText Markup Language(HTML) or other conventional forms, including embedded Extensible MarkupLanguage (XML), scripts, controls, and so forth. The computer system 100can also include a web server for generating and/or delivering the webpages to client computer systems.

In an exemplary embodiment, the computer system 100 can be provided as asingle unit, e.g., as a single server, as a single tower, containedwithin a single housing, etc. The single unit can be modular such thatvarious aspects thereof can be swapped in and out as needed for, e.g.,upgrade, replacement, maintenance, etc., without interruptingfunctionality of any other aspects of the system. The single unit canthus also be scalable with the ability to be added to as additionalmodules and/or additional functionality of existing modules are desiredand/or improved upon.

A computer system can also include any of a variety of other softwareand/or hardware components, including by way of non-limiting example,operating systems and database management systems. Although an exemplarycomputer system is depicted and described herein, it will be appreciatedthat this is for sake of generality and convenience. In otherembodiments, the computer system may differ in architecture andoperation from that shown and described here.

Robotic Surgical Systems

The systems, devices, and methods disclosed herein can be implementedusing a robotic surgical system. Various embodiments of robotic surgicalsystems are described in further detail in U.S. Pat. No. 8,831,782 filedJul. 15, 2013 entitled “Patient-Side Surgeon Interface For ATeleoperated Surgical Instrument,” Intl. Pat. Pub. No. WO2014151621filed Mar. 13, 2014 entitled “Hyperdexterous Surgical System,” Intl.Pat. Pub. No. WO2014151952 filed Mar. 13, 2014 entitled “Compact RoboticWrist,” and U.S. Pat. Pub. No. 2012/0158013 filed Dec. 17, 2010 entitled“Surgical System And Methods For Mimicked Motion,” which are herebyincorporated by reference in their entireties.

As will be appreciated by a person skilled in the art, electroniccommunication between various components of a robotic surgical systemcan be wired or wireless. A person skilled in the art will alsoappreciate that all electronic communication in the system can be wired,all electronic communication in the system can be wireless, or someportions of the system can be in wired communication and other portionsof the system can be in wireless communication.

FIG. 3 illustrates an embodiment of a robotic surgical system 200configured to be used by a user 202 (e.g., a surgeon, a surgicalassistant, etc.) during performance of a surgical procedure on a patient204. As in this illustrated embodiment, the robotic surgical system 200can include a controller 206, motors 208, and a movement mechanism 210.The controller 206 can be configured to receive an input from the user202 requesting movement, relative to the patient 204, of a surgicalinstrument coupled to the movement mechanism 210. The controller 206 canbe configured to cause the motors 208 to drive movement of the movementmechanism 210, thereby causing the movement of the surgical instrumentrequested by the user 202. Although the illustrated robotic surgicalsystem 200 includes a plurality of motors 208, a robotic surgical systemcan include a single motor. Similarly, although the illustrated roboticsurgical system 200 includes a single controller 206 and a singlemovement mechanism 210, a robotic surgical system can include aplurality of controllers and/or a plurality of movement mechanisms.

In an exemplary embodiment, the movement mechanism 210 can include anarm. The arm can be configured to move so as to cause movement of asurgical instrument coupled thereto in any one or more of the threetranslational directions (surge, heave, and sway) and in any one or moreof the three rotational directions (roll, pitch, and yaw) in response tocontrol by the controller 206. In an exemplary embodiment, the arm canbe configured to provide a plurality of degrees of freedom. More thansix degrees of freedom can be provided in a variety of ways, asmentioned above and as will be appreciated by a person skilled in theart. In general, the arm can include a mechanical member configured tomove in response to an input to the system 200 from the user 202. Theuser's input can be configured to cause the controller 206 to transmitan electronic signal to the motors 208 that causes the motors 208 toprovide a force (e.g., torque) to the arm, thereby causing movement ofthe arm. The arm can include a plurality of members jointed together,which can facilitate movement of the arm in a plurality of degrees offreedom via bending, twisting, etc. at various ones of the joints.

The arm can include an electromechanical arm. The electromechanical armcan include one or more mechanical members configured to move inresponse to an electronic input. Examples of mechanical members that canform the arm include elongate shafts, coupling mechanisms (e.g., clips,magnets, snap fit mechanisms, shaped members configured to seat aninstrument therein by interference fir or press fit, clamps, protrusionsconfigured to be seated in corresponding depressions formed in asurgical instrument, depressions configured to receive thereincorresponding protrusions extending from a surgical instrument, etc.)configured to removably and replaceably couple a surgical instrument tothe arm, and joints (e.g., hinges, gimbals, etc.).

FIGS. 4 and 5 illustrate an embodiment of an arm 300 in the form of anelectromechanical arm. The arm 300 is mounted to a surgical table 302using a frame 304 in the illustrated embodiment of FIG. 4, but the arm300 can be mounted to any of a variety of stationary items, a wall, atable, a cart, the ceiling, etc., in any of variety of ways to helpstabilize the arm 300 for use during a surgical procedure. The arm 300can include an active portion 300 a configured to be activelycontrolled, e.g., configured to move in response to electronic input,and a passive portion 300 b configured to be passively controlled, e.g.,configured to move in response to hand or other manual movement thereof.The passive portion 300 b can lack motors or other electrical features,while the active portion 300 a can include motors and other electricalfeatures, such as associated with the joints, to facilitate electroniccontrol thereof. In at least some embodiments, an arm can lack a passiveportion so as to be configured to be entirely actively controlled. Whilethe active and passive portions 300 a, 300 b are sometimes referred toherein as components of a single arm, a person skilled in the art willappreciate that the active portion 300 a and the passive portion 300 bcan be separate arms that are matable to each other.

The arm 300 can, as in this illustrated embodiment, include a pluralityof mechanical members 306, a plurality of joints 308, and a couplingmechanism 310. Adjacent ones of the mechanical members 306 can beattached together at one of joints 308. In this illustrated embodiment,the active portion 300 a of the arm 300 includes five mechanical members306 and four joints 308, the passive portion 300 b of the arm 300includes two mechanical members 306 and three joints 308, and the arm300 includes another joint 308 between the active and passive portions300 a, 300 b, but arms can have any number of mechanical members andassociated joints in its active and passive portions.

As shown in FIG. 5, the arm 300, e.g., the active portion 300 a thereof,can be configured to removably and replaceably couple to a surgicalinstrument 312 via the coupling mechanism 310. A distal end 314 of theinstrument 312 can be configured to be advanced into a body of apatient, e.g., through an incision, through a natural orifice, etc. Theinstrument's distal end 314 can thus include a working end of theinstrument 312 configured to facilitate performance of the surgicalprocedure within the patient. The instrument's distal end 314 caninclude an end effector, e.g., forceps/graspers, needle drivers,scissors, electrocautery hooks, staplers, clip appliers/removers,suction tools, irrigation tools, etc. As in this illustrated embodiment,the instrument 312 can be advanced into a patient's body through acannula 316 (e.g., a trocar, an introducer tube, etc.). The couplingmechanism 310 is shown in FIG. 5 coupled to the cannula 316, which hasthe surgical instrument 312 advanced therethrough.

Aspects of the arm 300 and the frame 304 are further described inpreviously mentioned Intl. Pat. Pub. No. WO2014151621 filed Mar. 13,2014 entitled “Hyperdexterous Surgical System” and Intl. Pat. Pub. No.WO2014151952 filed Mar. 13, 2014 entitled “Compact Robotic Wrist.”

FIG. 6 illustrates another embodiment of an arm 400 in the form of anelectromechanical arm. The arm 400 can generally be configured and usedsimilar to the arm 300 of FIGS. 4 and 5. The arm 400 can include aplurality of mechanical members 402, a plurality of joints betweenadjacent ones of the arms 402, and a coupling mechanism 404 configuredto removably and replaceably couple to a surgical instrument I. The arm400 includes five mechanical members 402 and four joints in thisillustrated embodiment, but as mentioned above, arms can have any numberof mechanical members and associated joints.

As shown in FIGS. 6 and 7, the arm 400 can be included in a roboticsurgical system 406 configured to facilitate performance of a surgicalprocedure on a patient P. FIG. 8 shows an example of the system 406 inuse. As in this illustrated embodiment, the system 406 can include auser interface sub-system 408 that can include at least one display 410configured to display information thereon to a user U, at least one userinput device 412 configured to receive a user input thereto to controlmovement of the arm 400, a visualization system 414 that can include atleast one display 416 configured to display thereon image(s) of asurgical procedure being performed using the system 406, a freelymovable user input device 418 (shown as pinchers in this illustratedembodiment) configured to receive a user input thereto to controlmovement of the arm 400 and configured to be freely moved around by theuser U (e.g., handheld and moved around any space in or near anoperating room, etc.), an additional arms 422 that can be configured andused similar to the arm 400, and a control system 426 configured tofacilitate control of the arms 400, 422 by translating user inputs tothe user input devices 412, 418, e.g., manual movement of a user inputdevice, movement indicated by touch on a touch screen, etc., to one orboth of the arms 400, 422 as appropriate. The system 406 in thisillustrated embodiment includes two arms 400, 422, but it can includeanother number of arms, e.g., three, four, etc. The at least one display410 of the user interface sub-system 408 can be configured as a userinput device, e.g., as a touchscreen configured to receive user touchinput thereon. The user interface sub-system 408 can be in the same roomas the patient P, or it can be in a different room.

The control system 426 can, as in this illustrated embodiment, includeat least one computer 428, one or more cables 430, and at least onepower supply 432. The computer 428 can include at least one processor(not shown). As mentioned above, at least some embodiments of controlsystems can be at least partially wireless, in which case at least someof the cables 430 need not be present. The robotic surgical system 406can include at least one foot pedal 434 coupled to the computer 428 viaone of the cables 430, which can allow the foot pedal 434 to serve as auser input device. The robotic surgical system 406 can include at leastone knee control (not shown) coupled to the computer 428 via one of thecables 430, similar to a knee control of a sewing machine, which canallow the knee control to serve as a user input device.

The robotic surgical system 406 can include a frame 424 for each of thearms 400, 422. The frames 424 in this illustrated embodiment are eachmounted to a surgical table 426, but as mentioned above, frames can bemounted elsewhere. The frame 424 in this illustrated embodiment includesa vertical extension movably coupled to a rail mounted to the table 426.The vertical extension can be configured to move along the rail, therebyfacilitating positioning of the arms 400, 422 relative to the patient P.

One or more manually operated surgical instruments 420, e.g.,instruments not under the control of the robotic surgical system 406,can be used to perform the surgical procedure being performed on thepatient P.

Aspects of the robotic surgical system 406 are further described inpreviously mentioned Intl. Pat. Pub. No. WO2014151621 filed Mar. 13,2014 entitled “Hyperdexterous Surgical System.”

FIG. 9 illustrates another embodiment of a robotic surgical system 500.As in this illustrated embodiment, the robotic surgical system 500 caninclude a display 502 and a control system 504 configured to be inelectronic communication with the display 502. The display 502 and thecontrol system 504 are in wired electronic communication in thisillustrated embodiment, but the electronic communication can bewireless. The control system 504 can include a computer system includinga display controller 506 configured to facilitate the display of imageson the display 502, such as images of tissue 508 visualized by anendoscope 510 coupled to the control system 504. The display 502 can becoupled to handles 512 a, 512 b configured to facilitate manual movementof the display 502, a hand-tracking transmitter 514 configured togenerate a field (e.g., an electromagnetic field, an optical field(e.g., light beams), etc.), a surgeon's viewer 516 (e.g., glasses, etc.)configured to facilitate three-dimensional (3-D) viewing of 3-D imagesshown on the display 502, and a boom 518 configured to mount the display502 to a stable surface (e.g., a wall, a table, etc.). The display 502can be configured to show two-dimensional (2-D) and/or 3-D images.

Movement of a user-controlled master tool (also referred to herein as amaster input device) 520, an embodiment of which is illustrated in FIG.10, in the field generated by the transmitter 514 can be configured toprovide sensed spatial position and orientation information in a 3-Dcoordinate system. The master tool 520 can be configured to transmit thespatial position and orientation information to the control system 504,such as by cables 522 a, 522 b. The control system 504, e.g., aprocessor thereof, can be configured to receive the transmitted spatialposition and orientation information and, in response thereto, cause aslave tool 524 to move in accordance with the user's movement of themaster tool 520. The robotic surgical system 500 can thus allow controlof the slave tool 524 via the master tool 520. The master tool 520 inthis illustrated embodiment includes first and second master tool grips520 a, 520 b that each include a plurality of levers 526, a plurality offinger loops 528, a palm rest 530, and a mode control button 532, butthe master tool 520 can have a variety of other configurations, as willbe appreciated by a person skilled in the art. The robotic surgicalsystem 500 can include any number of master tools and any number ofslave tools each configured to be controlled by the master tool(s).

One or more manually operated surgical instruments 534 can be used tomanipulate the tissue 508 in addition to the slave tool 524 that canmanipulate the tissue 508.

FIG. 9 illustrates first, second, third, and fourth coordinate systemsC1, C2, C3, C4 representing local coordinates that specify therespective position and orientation of the portion of the system 500with which they are associated. The first coordinate system C1 isassociated with the manually operated surgical instrument 534. Thesecond coordinate system C2 is associated with the slave tool 524. Thethird coordinate system C3 is associated with a user (not shown)visualizing the display 502, and hence also with the master input device520 configured to be manipulated by the user. The fourth coordinatesystem C4 is associated with the control system 506, and hence also withimages that the control system 506 causes to be displayed on the display502. In general, the control system 506 can be configured to map andtranslate the third coordinate system C3 into the second coordinatesystem C2, e.g., map and translate movement of the master tool 520 tomovement of the slave tool 524. For example, if the user is holding themaster input device 520, e.g., one of the first and second master toolgrips 520 a, 520 b, in one of his/her hands and moves that hand tohis/her right, thereby moving the held master input device 520 to theright, the control system 506 can be configured to correspondingly causea working end of the slave tool 524 to move to the right. This movementcan be accomplished by the control system 506 causing an arm to whichthe slave tool 524 is coupled, similar to the arms discussed herein, tomove. This movement of the slave tool 523 can “correct” for pivoting ofa trocar (not shown) through which the slave tool 524 may be inserted toaccess the tissue 508.

Aspects of the robotic surgical system 500 are further described inpreviously mentioned U.S. Pat. No. 8,831,782 filed Jul. 15, 2013entitled “Patient-Side Surgeon Interface For A Teleoperated SurgicalInstrument.”

As mentioned above, at least some of the methods, systems, and devicesfor controlling movement of a robotic surgical system provided hereincan be configured to facilitate movement of a surgical instrument into afield of view of a camera visualizing a surgical area. FIG. 11illustrates one embodiment of such a robotic surgical system 600. Therobotic surgical system 600 can generally be configured and used similarto the robotic surgical system 200 of FIG. 3. The robotic surgicalsystem 600 can include a computer system that includes a display device602, a controller 604, a plurality of movement mechanisms 606 a, 606 b,606N, and a notification mechanism 610 (e.g., an 10 interface such as adisplay, a speaker, a light, etc.). In at least some embodiments, thenotification mechanism 610 can be omitted, and the robotic surgicalsystem 600 can be configured to provide notifications using anotherelement thereof, as discussed further below. As in this illustratedembodiment, the movement mechanisms 606 a, 606 b, 606N can each includean electromechanical arm. The robotic surgical system 600 can include atleast two movement mechanisms 606 a, 606 b and, optionally, anadditional one or more movement mechanisms for a total of “N” movementmechanisms, where N is greater than or equal to three. Each of themovement mechanisms 606 a, 606 b, 606N can be configured to couple toone of a plurality of surgical instruments 608 a, 608 b, 608N, asdiscussed herein. The controller 604 can be configured to receive aninput from a user (not shown) requesting movement, relative to a patient(not shown), of a master one of the surgical instruments 608 a, 608 b,608N. The user can provide the input using a user input device 612(e.g., a master tool), as discussed herein. The controller 604 can beconfigured to cause motors (not shown) of the robotic surgical system600 to drive movement of the movement mechanisms 606 a, 606 b, 606N, asalso discussed herein, thereby causing movement of the surgicalinstruments 608 a, 608 b, 608N respectively coupled thereto.

In an exemplary embodiment, at least one of the surgical instruments 608a, 608 b, 608N can include a camera having a field of view andconfigured to visualize a surgical area such that the robotic surgicalsystem 600 can be coupled to at least one camera. To facilitatediscussion herein, the first surgical instrument 608 a in thisillustrated embodiment is presumed to include a camera.

As will be appreciated by a person skilled in the art, as illustratedfor example in FIG. 12, a camera (e.g., the first surgical instrument608 a) can define a field of view 614. The field of view 614 can includean area configured to be visible by the camera 608 a at any given timeas prescribed by the camera's configuration, e.g., by a configuration ofa lens of the camera 608 a and by a configuration of an image sensor ofthe camera 608 a. The area can be 2-D, or the area can be 3-D such thatthe area is a volume. As will be appreciated by a person skilled in theart, a camera may only show where something is along a line, with depthor distance being inferred.

The field of view 614 can be defined by a perimeter 614 p such that thearea visualized by the camera 608 a includes an area bounded by theperimeter 614 p, with the camera 608 a being unable to visualize thearea outside the perimeter 614 p. A specific size and shape of theperimeter 614 p can depend on the camera's configuration. As the cameramoves, the specific area of a target being visualized can change as theperimeter 614 p surrounds different portions of the target. For example,different portions of a surgical area can be visualized within theperimeter 614 p as the camera 608 a moves relative to the surgical areaor as the surgical area moves relative to the camera 608 a.

The display device 602 can be configured to display an image thereon(which can be a 2-D image or a 3-D image) corresponding to the areavisualized by the camera 608 a. The image can, as will be appreciated bya person skilled in the art, be an exact image of the visualized area orbe a modified image of the visualized area, e.g., be a two-dimensionalrepresentation of a three-dimensional area, be black and white insteadof color, have reference marks superimposed thereon, etc.

The robotic surgical system 600 can be configured to signal the user ofthe robotic surgical system 600 with a cue regarding a location of a oneof the surgical instruments 608 b, 608N relative to the field of view614 of the camera 608 a. The cue can be configured to help the userposition the one of the surgical instruments 608 b, 608N within thefield of view 614 such that the one of the surgical instruments 608 b,608N can be visualized by the user, e.g., in an image on the displaydevice 602. The controller 604 can be configured to extract a volumetricor surface model from the area within the field of view's perimeter 614p as visualized by the camera 608 a, which can be used by the controller604 in providing the cue via the notification mechanism 610.

The cue can have a variety of configurations. The cue can include anyone or more of an audible signal, a visual signal, and a tactile signal.Examples of an audible signal include a buzzing sound, a series ofbeeps, a speaking computerized voice, etc. Examples of a visual signalinclude a constantly illuminated light, a blinking light, an icon shownon the display device 602 and/or on another display device, text shownon the display device 602 and/or on another display device, etc.Examples of a tactile signal include a vibration (such as of a portionof the robotic surgical system 600, e.g., a vibration of the user inputdevice 612, etc.), a temperature change (such as heating or cooling of aportion of the robotic surgical system 600, e.g., a temperature changeof at least a portion of the user input device 612, etc.), etc.Providing more than one type of cue, e.g., an auditory signal and avisual signal, a tactile signal and an audible signal, etc. may helpensure that the user receives the cue. For example, providing an audiblesignal with at least one other type of signal may help ensure that theuser receives the cue if the user is being subjected to other noise(e.g., instrument motors, talking medical personnel, etc.) that maydrown out the audible signal. For another example, providing a visualsignal with at least one other type of signal may help ensure that theuser receives the cue if the visual signal is provided outside theuser's current line of sight. For yet another example, providing acolor-based visual signal with at least one other type of signal mayhelp ensure that a colorblind user who may have difficulty with thevisual signal receives at least one of a non-color dependent audiblesignal and non-color dependent tactile signal.

The controller 604 can be configured to cause the notification mechanism610 to provide the cue, e.g., by transmitting a signal thereto. In atleast some embodiments, the notification mechanism 610 can include aplurality of notification mechanisms with each one of the notificationmechanisms being coupled to one of the arms 608 a, 608 b, 608N, such aseach of the arms 608 a, 608 b, 608N having a light mounted thereon, etc.As mentioned above, in embodiments in which the notification mechanism610 is omitted, another portion of the robotic surgical system 600 canbe configured to provide the cue, such as the display device 602 beingconfigured to display the cue thereon.

The controller 604 can be configured to cause the notification mechanism610 to constantly provide the cue, e.g., constantly illuminate a light,constantly display a visible icon, text, etc. on the display device 602,constantly sound an audible signal, etc., as long as the one of thesurgical instruments 608 b, 608N whose location relative to the field ofview 614 is at issue is not located within the field of view 614. Theconstant providing of the cue can include the cue being configured inits normal state to repeatedly turn on and off, such as in the case of ablinking light or a series of audible beeps. The controller 604 can beconfigured to stop providing the cue when the one of the surgicalinstruments 608 b, 608N whose location relative to the field of view 614is at issue is located at least partially within the field of view 614.Constant provision of the cue may thus help the user controllingmovement of the one of the surgical instruments 608 b, 608N whoselocation relative to the field of view 614 is at issue, e.g., the userhandling the user input device 612, to determine when that surgicalinstrument has entered the field of view 614 because the cue hasstopped.

The cue can be configured to indicate to the user whether the currentmovement of the surgical instrument at issue will cause the instrumentto enter the field of view 614. The controller 604 can be configured tochange the cue during provision of the cue, thereby providing theinformation to the user regarding whether the instrument's currentmovement will cause the instrument to enter the field of view 614. Thechange in the cue may facilitate the user's determination of how, orwhether, to continue the instrument's movement, e.g., in continuing tomanipulate and provide input to the user input device 612. In the caseof an audible signal, the cue can change in one or more ways including,e.g., changing an amplitude of the sound, changing a pitch of the sound,changing a loudness of the sound, changing a modulation of the sound,changing a chord (major versus minor), using binaural displacement,providing tones more or less frequently, etc. In the case of a visualsignal, the cue can change in one or more ways including, e.g., changingphysical location of the visual cue on the display screen on which it isdisplayed, changing color, changing brightness, changing hue, changingsaturation, using temporal modulation, blinking a light more or lessfrequently, etc. In the case of a tactile signal, the cue can change inone or more ways including, e.g., changing vibration strength, becomingcolder, becoming hotter, etc.

A first type of change in the cue can indicate that the instrument'scurrent movement is moving the instrument toward being in the field ofview 614, and a second type of change in the cue can indicate that theinstrument's current movement is moving the instrument away from beingin the field of view 614. For example, for an audible signal, the firsttype can include the sound becoming quieter and the second type caninclude the sound becoming louder. For another example, for an audiblesignal, the first type can include beeps being sounded more frequently(e.g., with less “dead” space between the beeps) and the second type caninclude beeps being sounded less frequently (e.g., with more “dead”space between the beeps). For yet another example, for an audiblesignal, the first type can include a higher pitched sound and the secondtype can include a lower pitched sound. For another example, for avisual signal, the first type can include the cue becoming brighter incolor and the second type can include the cue becoming duller in color.For still another example, for a visual signal, the first type caninclude a light blinking faster and the second type can include thelight blinking slower. For another example, for a visual signal, thefirst type can include a light transitioning toward one color and thesecond type can include the light transitioning toward another color.For yet another example, for a tactile signal, the first type caninclude a faster vibration and the second type can include a slowervibration. For another example, for a tactile signal, the first type caninclude cooling and the second type can include heating. For yet anotherexample, for a tactile signal, the first type can include a temperaturechange toward an ambient temperature and the second type can include adecrease in temperature.

The change can represent a single parameter or a plurality ofparameters. One example of the single parameter includes a simplebetter/worse signal with the first type of change indicating that theinstrument's current movement is moving the instrument toward being inthe field of view 614 (“better”) and a second type of change indicatingthat the instrument's current movement is moving the instrument awayfrom being in the field of view 614 (“worse”). The change in the cueindicating the better/worse parameter may facilitate the user guidingthe instrument in a way to achieve the “better” cue. Another example ofthe single parameter includes an indication as to whether a currentdirection of the instrument is leading the instrument toward being inthe field of view 614 or is moving the instrument away from being in thefield of view 614. In other words, the cue can be configured to indicateto the user whether the instrument's current bearing should be adjustedin order to better aim the instrument toward the field of view 614.Another example of the single parameter includes an indication as to howmuch distance remains between the surgical instrument and the field ofview 614. The change in the cue indicating the direction parameter mayfacilitate user decision-making in controlling movement of theinstrument, such as the user deciding to move the instrument slower asthe instrument approaches the field of view 614, the user deciding tomake more radical trajectory adjustments of the instrument the fartherthe instrument is from entering the field of view 614, etc. The changerepresenting a plurality of parameters can include multiple ones of thesingle parameters, e.g., better/worse and direction; better/worse,direction, and remaining distance; etc.

FIG. 13 illustrates one embodiment of a process 700 of the controller604 providing a cue based on a location of one of the instruments 608 b,608N relative to the field of view 614, e.g., based on the direction anddistance from the one of the instruments 608 b, 608N (such as an endeffector thereof) to the nearest point on the field of view 614, whetherthe field of view 614 be 2-D or 3-D. For ease of discussion of FIG. 13,the second instrument 608 b is considered to be the instrument at issue.The process 700 can start 702 in any of a variety of ways. One exampleof the start 702 includes the user providing an input to the roboticsurgical system 600 indicating that the user desires to track thelocation of the second instrument 608 b relative to the field of view614, e.g., the user input device 612 can include a tracking controlmechanism (not shown), e.g., a button, lever, etc., configured to bemanipulated by the user similar to the mode control button discussedherein; the display device 602 can be configured to receive a touchinput thereto; an IO interface (not shown), such as a keyboard, coupledto the controller 604 can be configured to receive user input thereto;etc. The start 702 can thus be on-demand. Another example of the start702 includes when the second surgical instrument 608 b moves from beingwithin the field of view 614 to outside the field of view 614. The start702 can thus be automatic. Yet another example of the start 702 includesthe second surgical instrument 608 b being advanced into a trocar (notshown) through which the second surgical instrument 608 b gains accessto a patient's body, which is also an example of the start 702 beingautomatic.

After the start 702 of the process 700, the controller 604 can beconfigured to determine 704 a current position of the surgicalinstrument 608 b at issue relative to the field of view 614. The currentposition can be determined in a variety of ways. For example, theposition of the instrument 608 b can be based on a trajectory of theinstrument 608 b as defined by a longitudinal axis of the instrument 608b, e.g., a longitudinal axis of an elongate shaft of the instrument 608b, etc., since the instrument's longitudinal axis can indicate adirection of the instrument's movement. For another example, theposition of the instrument 608 b can be based on a trajectory of theinstrument 608 b as defined by a longitudinal axis of a trocar throughwhich the instrument 608 b is being advanced, since the instrument'smovement can be limited by the trocar, e.g., by a cannulated elongateportion of the trocar through which the instrument 608 b is beingadvanced. For yet another example, the position of the instrument 608 bcan be based on a position of the second arm 606 b to which the secondsurgical instrument 608 b is coupled, since the instrument's movement iscontrolled via the second arm 606 b. The controller 604 can beconfigured to provide 706 a cue based on the determined currentposition.

The process 700 can be iterative as long as the surgical instrument 608b is not within the field of view 614. After providing the cue, thecontroller 604 can be configured to determine 708 whether the instrument608 b is within the field of view 608 b, e.g., by analyzing the cameraimage of the field of view 614, by comparing a coordinate of theinstrument's current position with a coordinate map corresponding to thefield of view 614, etc. If the instrument 608 b is determined 708 to bewithin the field of view 614, the controller 604 can be configured tostop 710 providing the cue. If the instrument 608 b is determined 708 tonot be within the field of view 614, the controller 604 can beconfigured to continue providing the cue unless the controller 604receives 712 a signal to stop providing 706 the cue. The signal to stop710 providing the cue can be input by the user similar to the user beingable to start 702 the process 700. The user may want to stop the cue if,for example, the instrument 608 b will be remaining in its currentposition while others of the instruments 608 a, 608N are manipulatedand/or while other aspects of the surgical procedure are beingperformed. If a signal to stop providing 706 the cue is not received712, then the controller 704 can be configured to again determine 704 acurrent position of the instrument 608 b relative to the field of view614 and continue providing 706 the cue, with the cue being changed asneeded based on the most recently determined 704 current position of theinstrument 608 b.

FIG. 14 illustrates another embodiment of a process 800 of thecontroller 604 providing a cue based on a location of one of theinstruments 608 b, 608N relative to the field of view 614. The process800 of FIG. 14 generally corresponds to the process 700 of FIG. 13. Inthe illustrated embodiment of FIG. 14, the process 800 involvesadvancement of the instrument 608 b at issue through a trocar. Theprocess 800 of FIG. 14 can start 802 by the user deciding to employ ahidden one of the surgical instruments 608 b, 608N, which for ease ofdiscussion of FIG. 14, will be considered to be the second surgicalinstrument 608 b. The user can signal this decision to the controller604 in any of a variety of ways, e.g., actuating the tracking controlmechanism, inserting the instrument 608 b into a trocar, etc. Thecontroller 604 can be configured to compute 804 a bearing of a linebetween the trocar and the target, e.g., the field of view 614. The linecan include a trajectory line of the trocar, which can, as discussedherein, be defined by a longitudinal axis of the trocar. The computing804 can include calculating a direction of a line between a pivot point(e.g., virtual center, insertion point, etc.) of the trocar and areference point in the field of view 614. The line may, but usually willnot coincide with the longitudinal axis of the trocar until the usermakes it so by providing an input to the system. The controller 604 canbe configured to compute 806 a tool bearing error indicative of theinstrument's current offset from intersection with the field of view614, e.g., a current offset of the bearing of the line from the field ofview 614 as defined by its perimeter 614 p. For example, the toolbearing error can be based on an angular offset of the bearing of theline from a centerpoint of the field of view 614 as defined by theperimeter 614 p. In embodiments in which the bearing of the line isdetermined based on a trajectory line of the instrument 608 b as definedby a longitudinal axis of the instrument 608 b, the tool bearing errorcan be similarly computed. For another example, the tool bearing errorcan be based on an angular offset of the bearing of the line from acenterpoint of a line joining a pair of instrument working ends visiblewithin the field of view 614. For yet another example, the tool bearingerror can be based on an angular offset of the bearing of the line froma point along the camera's trajectory that is nearest to a working endof any surgical instrument present within the field of view 614. For yetanother example, the tool bearing error can be based on an angularoffset of the bearing of the line from a first intersection of the linewith a surface within the field of view 614.

The controller 604 can be configured to signal 808 the computed bearingerror by providing a cue. If the movement of the instrument 608 b hasmoved the instrument to be in the field of view 614, e.g., if thecontroller 604 determines 810 that the instrument 608 b is within thefield of view 614, then the controller 604 can be configured toterminate 812 the provided signal. The surgical procedure can thencontinue 814. If the movement of the instrument 608 b has not moved theinstrument to be in the field of view 614, then the controller 604 canbe configured to again compute 806 the tool bearing error and againsignal 808 the computed bearing error, with the cue being changed ifneeded based on the re-computed tool bearing error. The process 800 canthus be iterative.

FIGS. 15-17 illustrate one embodiment of a performance of a surgicalprocedure in which a robotic surgical system can be configured toprovide a cue based on a location of a surgical instrument relative to afield of view of a camera. The performance of the surgical procedure isdescribed with respect to the robotic surgical system 600 of FIG. 11,but this and other embodiments of performing a surgical procedure can beperformed using other robotic surgical systems described herein. Thesurgical procedure can begin by preparing the patient for surgery andmaking one or more appropriately sized incisions at a desired location,as will be appreciated by a person skilled in the art. The surgicalprocedure can, as in this illustrated embodiment, include a minimallyinvasive procedure in which one or more access devices, trocars in thisillustrated embodiment, can be positioned in any one or more of theincision(s) to provide access to the surgical site. In otherembodiments, the surgical procedure can be an open procedure.

To facilitate discussion herein of this illustrated embodiment in FIGS.15-17, the plurality of surgical instruments coupled to the roboticsurgical system 600 via the “N” number of arms is presumed to includefour surgical instruments, with a fourth one of the surgical instruments608 d being presumed to include a camera. The robotic surgical system600 is accordingly presumed for purposes of discussing this illustratedembodiment to include at least four arms, with each of the plurality ofsurgical instruments 608 a, 608 b, 608 c, 608 d being coupled to one ofthe arms. The camera 608 d can have a line of sight (LOS), e.g., acentral axis of a 3-D field of view, and can define a field of view 616.FIGS. 15-17 illustrate the surgical instruments 608 a, 608 b, 608 c, 608d relative to an insufflated patient of substantial curvature, convexout of the page.

Once the patient is prepared for surgery, each of the plurality ofsurgical instruments 608 a, 608 b, 608 c, 608 d can be positionedrelative to a patient in any of a variety of ways, as will beappreciated by a person skilled in the art. As shown in FIG. 15, thecamera 608 d can be positioned such that its field of view 616 surroundsa target surgical area 618, also referred to herein as a workzone. Inother words, the workzone 618 can be contained with the field of view'sperimeter 616 p. As also shown in FIG. 15, the first surgical instrument608 a can be advanced through a first trocar 620 a inserted into thepatient such that a distal end 622 a of the first surgical instrument608 a (e.g., a working end of the first surgical instrument 608 a) islocated at a desired location, such as within the workzone 618 as inthis illustrated embodiment, the second surgical instrument 608 b can beadvanced through a second trocar 620 b inserted into the patient suchthat a distal end 622 b of the second surgical instrument 608 b (e.g., aworking end of the second surgical instrument 608 b) is located at adesired location, such as within the workzone 618 as in this illustratedembodiment, and the third surgical instrument 608 c can be advancedthrough a third trocar 620 c inserted into the patient such that adistal end 622 c of the third surgical instrument 608 c (e.g., a workingend of the third surgical instrument 608 c) is located at a desiredlocation, such as within the workzone 618 as in this illustratedembodiment.

FIG. 16 illustrates the surgical procedure at a time subsequent to thetime of FIG. 15. In FIG. 16, surgical attention has shifted downward andto the left as reflected by the workzone 618′ being shifted in positiondownward and to the left of the workzone 618 of FIG. 15. The locationsthrough which the surgical instruments 608 a, 608 b, 608 c, 608 d havebeen inserted into the patient, e.g., incisions through which theinstruments have been directly inserted (as with the camera 608 d) ortrocars 620 a, 620 b, 620 c positioned in incisions and through whichthe instruments have been inserted (as with the first, second, and thirdinstruments 608 a, 608 b, 608 c), remain the same as in FIG. 15. Thecamera 608 d has been angularly adjusted in position such that its fieldof view 616 surrounds the shifted workzone 618′, and the first andsecond instruments 608 a, 608 b have been angularly adjusted withintheir respective trocars 620 a, 620 b such that their respective distalends 622 a, 622 b are positioned to desired locations within the shiftedworkzone 618′. The third surgical instrument 608 c, shown by the dottedline in FIG. 16, is at the same position as in FIG. 15 with its distalend 622 c located outside the camera's field of view 616.

The third surgical instrument 608 c can be angularly adjusted at anangle α such that the surgical instrument 608 c′, represented by a solidline in FIG. 16, is directed toward the field of view 616. As the thirdsurgical instrument 608 c is moved from its position in FIG. 15 inresponse to a user's input to the user input device 612, the controller604 can be configured to cause the notification mechanism 610 to providea cue to the user indicative of the third surgical instrument's locationrelative to the field of view 616 in its shifted position shown in FIG.16. As discussed herein, the controller 604 can be signaled to beginproviding the cue by the user providing an input to the robotic surgicalsystem 600 indicating that the user desires to track the location of thethird instrument 608 c relative to the field of view 616, or thecontroller 604 be configured to automatically begin providing the cue inresponse to the third surgical instrument 608 c be adjusted in positionby the user. As also discussed herein, the controller 604 can beconfigured to provide the cue based on a trajectory 624 of the thirdsurgical instrument 608 c moving toward an adjusted trajectory 624′ thataligns the third surgical instrument 608 c to intersect with the fieldof view 616 and the shifted workzone 618′. The third instrument'strajectory in this illustrated embodiment is based on a longitudinalaxis of the third trocar 620 c through which the instrument 608 c isadvanced, although as mentioned herein, the trajectory can be based on alongitudinal axis of the third surgical instrument 608 c. The controller604 can thus provide the cue to the user in real time with the thirdsurgical instrument's movement from its initial trajectory 624 so as tofacilitate positioning of the third surgical instrument 608 c at theadjusted trajectory 624′ along which the third surgical instrument 608 ccan be advanced to enter the camera's field of view 616 and hence beable to be visualized by the user. For example, in beginning to move thethird surgical instrument 608 d from its position in FIG. 15, the usercan cause the third surgical instrument 608 d to move with a quickcircular motion, which can help the user quickly discover the adjustedtrajectory 624′ based on the changing cue during the circular motion.The user can then more slowly move the third surgical instrument 608 cto more finely find the adjusted trajectory 624′ and move the thirdsurgical instrument 608 c therealong to move the third surgicalinstrument's distal end 622 c′ in to view within the field of view 616.For another example, the user can slowly move the third surgicalinstrument 608 c from its position in FIG. 15 in a side-to-side sweepingmotion to help locate the adjusted trajectory 624′.

Once the distal end 622 c′ of the third surgical instrument 608 c entersthe field of view 616, the controller 604 can be configured to stop thecue, as discussed herein.

FIG. 17 illustrates the surgical procedure at a time subsequent to thetime of FIG. 16. In FIG. 17, the workzone 618″ has shifted in positionupward and to the right of the workzone 618′ of FIG. 16. The locationsthrough which the surgical instruments 608 a, 608 b, 608 c, 608 d havebeen inserted into the patient remain the same as in FIGS. 15 and 16.The camera 608 d has been angularly adjusted in position such that itsfield of view 616 surrounds the second shifted workzone 618″, and thefirst and second instruments 608 a, 608 b have been angularly adjustedwithin their respective trocars 620 a, 620 b such that their respectivedistal ends 622 a, 622 b are positioned to desired locations within thesecond shifted workzone 618″. The third surgical instrument 608 c, shownby the solid line in FIG. 17, is at the same position as in FIG. 16 withits distal end 622 c′ located outside the camera's field of view 616.The third surgical instrument 608 c can be angularly adjusted at anangle θ such that the surgical instrument 608 c″, represented by adotted line in FIG. 17, is directed toward the field of view 616.Similar to that discussed above regarding claim 16, as the thirdsurgical instrument 608 c is moved from its position in FIG. 16 inresponse to a user's input to the user input device 612, the controller604 can be configured to cause the notification mechanism 610 to providea cue to the user indicative of the third surgical instrument's locationrelative to the field of view 616 in its shifted position shown in FIG.17. Once the distal end 622 c″ of the third surgical instrument 608 denters the field of view 616, the controller 604 can be configured tostop the cue, as discussed herein.

As mentioned above, at least some of the methods, systems, and devicesfor controlling movement of a robotic surgical system provided hereincan be configured to facilitate detection of collision of a portion ofthe robotic surgical system with another object. FIG. 18 illustrates oneembodiment of such a robotic surgical system 900. The robotic surgicalsystem 900 can generally be configured and used similar to the roboticsurgical system 200 of FIG. 3. The robotic surgical system 900 caninclude a computer system that includes a display device (not shown), acontroller 904, a plurality of movement mechanisms 906 a, 906N, and anotification mechanism 902 (e.g., an IO interface such as a display, aspeaker, a light, etc.). In at least some embodiments, the notificationmechanism 902 can be omitted, and the robotic surgical system 900 can beconfigured to provide notifications using another element thereof, asdiscussed further below. As in this illustrated embodiment, the movementmechanisms 906 a, 906N can each include an electromechanical arm. Therobotic surgical system 900 can include at least one movement mechanism906 a and, optionally, an additional one or more movement mechanisms fora total of “N” movement mechanisms, where N is greater than or equal totwo. Each of the movement mechanisms 906 a, 906N can be configured tocouple to one of a plurality of surgical instruments (not shown), asdiscussed herein. The controller 904 can be configured to receive aninput from a user (not shown) requesting movement, relative to a patient(not shown), of a master one of the surgical instruments 906 a, 906N.The user can provide the input using a user input device (not shown)(e.g., a master input device), as discussed herein. The controller 904can be configured to cause motors (not shown) of the robotic surgicalsystem 900 to drive movement of the movement mechanisms 906 a, 906N, asalso discussed herein, thereby causing movement of the surgicalinstruments respectively coupled thereto.

As in this illustrated embodiment, each of the movement mechanisms 906,906N can have at least one sensor 908 a, 908N attached thereto. Thesensors 908 a, 908N can be configured to facilitate the detection of acollision of the one of the movements mechanisms 908 a, 908 b to whichit is coupled with another object before the collision occurs. Thesensors 908 a, 908N can be configured to be in electronic communicationwith the controller 904. In this illustrated embodiment, each of themovement mechanisms 906 a, 906N has a sensor attached thereto. In otherembodiments, at least one but less than all of the “N” movementmechanisms 906 a, 906N can have a sensor attached thereto, which mayhelp reduce monetary cost of the system 900, may reduce processingdemands on the controller 904, and/or may account for at least one ofthe “N” movement mechanisms being unlikely to move during performance ofa surgical procedure (e.g., if the movement mechanism will have asurgical instrument coupled thereto that is unlikely to move duringperformance of a surgical procedure).

The sensors 908 a, 908N can be attached to their respective movementmechanisms 906 a, 906N in any of a variety of ways. In at least someembodiments, the sensors 908 a, 908N can be directly attached to theirrespective movement mechanisms 906 a, 906N, such as with magnet(s), withadhesive, by being embedded therein, etc. In at least some embodiments,the sensors 908 a, 908N can be directly attached to their respectivemovement mechanisms 906 a, 906N by being attached to a casing of themovement mechanism, such as a shroud or a drape. The casing can beconfigured to serve a cosmetic purpose (e.g., improve aesthetics of thesystem 900), to facilitate branding (e.g., by having a hospital name,manufacturer name, etc. printed thereon), and/or to serve a functionalpurpose (e.g., facilitate cleaning of the system 900 by allowing removalof the casing for replacement with either the same or a different casingfor subsequent use of the system 900, carry electrical conductor(s),carry fluid conductor(s), etc.). In embodiments in which one or more ofthe sensors 909 a, 908N are attached to a casing instead of directly toa movement mechanism, the detected collision may be between the casingand another object (e.g., another casing, another one of the arms,medical personnel, etc.) but is considered herein to be between theother object and the movement mechanism that includes the casing.

The sensors 908 a, 908N can have any of a variety of configurations. Insome embodiments, each of the sensors 908 a, 908N can be of a same typeas each other, which may facilitate assembly of the system 900 and/ormay facilitate programming and/or processing of the controller 904. Inother embodiments, one or more of the sensors 908 a, 908N can have atype different from any of the others of the sensors 908 a, 908N, whichmay allow collisions to be detected in two or more different ways andthereby help increase chances of a collision being prevented.

One type of sensor for the sensors 908 a, 908N includes an energy probeconfigured to probe space with energy. One type of energy probe includesa light probe. The light probe can be passive, in which the sensor isconfigured to look for light from other sources, or active, in which thesensor is configured to look for light from its own source.

A passive light probe can be configured to sense light emitted bysources, such as other light probes. The passive light probe can also beconfigured to emit light configured to be sensed by other light probes.The light configured to be sensed by the light probe and the lightconfigured to be emitted by the light probe can be light visible by thehuman eye or can be light, such as infrared light, that is not visibleto the human eye. The light can have any of a variety of wavelengths andcan be selected for compatibility with transmission windows in thesensor materials. The more light sources and detectors used may helpincrease a confidence in detected light and/or increase chances ofdetecting an impending collision. The fewer light sources and detectorsused, the more cost effective the system may be. Each of the lightprobes can be associated with a light protection zone, with energyemitted and collected through the protection zone using diffusingfibers. Alternatively, the light probes as microstructured reflectorarrays may distribute light generally away from the surface to which itis attached when illuminated by a beam parallel thereto, with returnedor received light being collected and directed to a collocated detectorsimilar to that which may occur with diffusing fibers.

The controller 904 can be configured to determine from the light sensedby the passive light probe whether a collision is impending. Forexample, when the light detected by the sensor is equal to or greaterthan a predetermined threshold amount of light, the controller 904 canbe configured to determine that a possible collision is imminent. Inother words, when a light sensed by the sensor is bright enough, thesource of the sensed emitted light can be considered to be close to thearm to which the sensor is attached to indicate a possible impendingcollision. For another example, when the light detected by the sensor isincreasing at a rate that is equal to or greater than a predeterminedrate of increase, the controller 904 can be configured to determine thata possible collision is imminent. In other words, the source of thesensed emitted light can be considered to be becoming closer to the armhaving the light probe attached thereto (by movement of the light sourceand/or by movement of the arm having the sensor attached thereto) suchthat a collision may be imminent. For yet another example, when thelight detected by the sensor is increasing at a rate that is equal to orgreater than a predetermined rate of increase and the light sensed bythe sensor is bright enough, the controller 904 can be configured todetermine that a possible collision is imminent.

An active light probe can be configured similar to a passive light probebut instead of being configured to sense light emitted by other lightprobes can be configured to sense its own light as reflected backthereto. The controller 904 can be configured to determine from thelight sensed by the active light probe whether a collision is impending.For example, when the light detected by the sensor is equal to orgreater than a predetermined threshold amount, the controller 904 can beconfigured to determine that a possible collision is imminent. In otherwords, when a light sensed by the sensor is bright enough, the objectthat reflected the light back to the sensor can be considered to beclose to the arm to which the sensor is attached to indicate a possibleimpending collision. For another example, when the light detected by thesensor is increasing at a rate that is equal to or greater than apredetermined rate of increase, the controller 904 can be configured todetermine that a possible collision is imminent. In other words, theobject that reflected the light back to the sensor can be considered tobe becoming closer to the arm having the light probe attached thereto(by movement of the light source and/or by movement of the arm havingthe sensor attached thereto) such that a collision may be imminent. Foryet another example, when the light detected by the sensor is increasingat a rate that is equal to or greater than a predetermined rate ofincrease and the light sensed by the sensor is bright enough, thecontroller 904 can be configured to determine that a possible collisionis imminent.

Another type of energy probe for the sensors 908 a, 908N includes anelectrical probe. In at least some embodiments, the electrical probe caninclude a transmission line. For example, the electrical probe can beconfigured to radiate energy from a leaky transmission line routed alonga movement mechanism or along the casing of the movement mechanism, andthe electrical probe can be configured to observe driving pointcharacteristics of the leaky transmission line. The controller 904 canbe configured to use time-domain reflectometry (TDR) to determine, basedon the observed driving point characteristics, whether a possiblecollision is imminent. The controller 904 may accomplish thisdetermination, for example, by considering a possible collision to beimminent when a magnitude of the detected reflections is equal to orgreater than a predetermined threshold amount. For another example ofthe electrical probe configured as a transmission line, the electricalprobe can include an array of low-efficiency antenna. The controller 904can be configured to determine a standing wave ratio (SWR) of thetransmission line, as the SWR will vary as objects are at differentdistances from the sensor. When the SWR is determined to be equal to orgreater than a predetermined threshold amount, the controller 904 candetermine that a possible collision is imminent.

In at least some embodiments, the electrical probe can include aconductive layer with respect to a common ground, such as a casing of amovement mechanism having a conductive outer layer with respect to acommon ground. The controller 904 can be configured to determine currentin the sensor's drive electrode, synchronous to the sensor's own signal(e.g., I and Q components thereof). Capacitance increases withdecreasing distance, so the current flow will change accordingly. Thus,based on the determined current, the controller 904 can be configured todetermine whether a possible collision is imminent. When the determinedcurrent has changed by more than a predetermined threshold amount, thecontroller 904 can determine that a possible collision is imminent. Inother words, the higher the capacitance, the closer the nearby objectand thus the more likely a collision may occur.

Another type of energy probe for the sensors 908 a, 908N includes asonic probe. The sonic probe can be passive, in which the sensor isconfigured to listen for sound from other sources, or active, in whichthe sensor is configured to listen for sound from its own source.

A passive sonic probe can be configured to sense sound emitted bysources, such as other sonic probes. The passive sonic probe can also beconfigured to emit sound configured to be sensed by other sonic probes.The sound configured to be sensed by the sonic probe can be a sound thatis inaudible to a human, which may help prevent the sound fromdistracting medical personnel performing a surgical procedure. Sounds atsubaudible frequencies can have a wavelength that is long enough for asingle transducer to be used and can have a nearly omnidirectionalradiation pattern. At superaudible frequencies, the movement mechanismand/or the casing thereon may be multiple wavelengths long. A singletransducer may thus not be effective for these larger size systems.Multiple transducers may thus be used to account for the multiplewavelength size of an arm and/or casing by having phases and amplitudesto provide a radiating and sending pattern that covers the entire size.

The controller 904 can be configured to determine from the sound sensedby the passive sonic probe whether a collision is impending. Forexample, when the sound detected by the sensor is equal to or greaterthan a predetermined threshold amount of sound, the controller 904 canbe configured to determine that a possible collision is imminent. Inother words, when a sound sensed by the sensor is loud enough, thesource of the sensed emitted sound can be considered to be close to thearm to which the sensor is attached to indicate a possible impendingcollision. For another example, when the sound detected by the sensor isincreasing at a rate that is equal to or greater than a predeterminedrate of increase, the controller 904 can be configured to determine thata possible collision is imminent. In other words, the source of thesensed emitted sound can be considered to be becoming closer to the armhaving the sonic probe attached thereto (by movement of the sound sourceand/or by movement of the arm having the sensor attached thereto) suchthat a collision may be imminent. For yet another example, when thesound detected by the sensor is increasing at a rate that is equal to orgreater than a predetermined rate of increase and the sound sensed bythe sensor is loud enough or has a high enough dB level, the controller904 can be configured to determine that a possible collision isimminent.

An active sonic probe can be configured similar to a passive sonic probebut instead of being configured to sense sound emitted by other soundprobes can be configured to sense its own sound as reflected backthereto. The controller 904 can be configured to determine from thesound sensed by the active sonic probe whether a collision is impending.For example, when the sound detected by the sensor is equal to orgreater than a predetermined threshold amount, the controller 904 can beconfigured to determine that a possible collision is imminent. In otherwords, when a sound sensed by the sensor is loud enough or has a highenough dB level, the object that reflected the sound back to the sensorcan be considered to be close to the arm to which the sensor is attachedto indicate a possible impending collision. For another example, whenthe sound detected by the sensor is increasing at a rate that is equalto or greater than a predetermined rate of increase, the controller 904can be configured to determine that a possible collision is imminent. Inother words, the object that reflected the sound back to the sensor canbe considered to be becoming closer to the arm having the sonic probeattached thereto (by movement of the sound source and/or by movement ofthe arm having the sensor attached thereto) such that a collision may beimminent. For yet another example, when the sound detected by the sensoris increasing at a rate that is equal to or greater than a predeterminedrate of increase and the sound sensed by the sensor is loud enough orhas a high enough dB level, the controller 904 can be configured todetermine that a possible collision is imminent.

Frequency has an inverse relationship to wavelength, so sounds beingdetected or emitted at higher frequencies have shorter wavelengths. Itis thus more likely at higher frequencies that multiple transducers maybe needed to account for the multiple wavelength size of an arm and/orcasing when passive sonic probes are used. Thus, for higher frequencies,active sonic probes may be desired over passive sonic probes since alower number of sonic probes may be needed.

Regardless of the type of energy probe used, the controller 904 can beconfigured to coordinate operations to reduce interference from thedifferent energies involved in the case of multiple energy probes inuse. For example, frequency channels, codes, or time slots could beassigned to different ones of the energy probes, and the controller 904can thus be able to identify the various energy probes and energiessensed thereby. In at least some embodiments, the energy probe can beconfigured to include its own processing capability, such as by a sensorbeing included on a microchip that also includes a microprocessor, suchthat the energy probe can be configured to communicate sensed dataand/or collision determination to the controller 904 only when themicroprocessor determines that a possible collision is imminent. Forexample, the microprocessor can, similar to radio communications, beconfigured to use spread spectrum techniques, token passing,listen-before-talk, etc.

Another type of sensor for the sensors 908 a, 908N includes a matterprobe configured to probe space with matter. The matter probe contactinganother object can be indicative of a possible imminent collision, e.g.,possible imminent contact of the object with the movement mechanism towhich the matter probe is attached. The matter probe can be configuredto not disrupt overall functioning of the robotic surgical system 900.For example, the matter that probes space can be relatively small and/orexert a force that is not high enough to disrupt the overall functioningof the robotic surgical system 900.

One type of matter probe includes a mechanism extension that extendsfrom a surface to which it are attached, e.g., from a surface of amovement mechanism or a casing thereof. The mechanism extension can bein the form of an elongate whisker that can be attached to the surfacein any of a variety of ways, e.g., with adhesive, by plugging into asocket on the surface, etc. The whisker can have only one end thereofattached to the surface, or the whisker can have both its ends attachedto the surface so as to form an arch. In an exemplary embodiment, aplurality of mechanism extensions can be attached to a movementmechanism and can extend in different directions therefrom, therebyhelping to prevent a collision from occurring in any direction in whichthe movement mechanism moves or in any direction that an objectapproaches the movement mechanism.

The mechanism extension can have a variety of configurations. Themechanism extension can be flex-sensitive, such as through resistive,piezo, or optical sensing. The mechanism extension can be static, or,the mechanism extension can be configured to microscopically vibrate.The vibration may increase chances of contact of the mechanism extensionwith an object near the movement arm to which the mechanism extension isattached. The mechanism extension can be a standalone element, or themechanism extension can be part of an array or strip including aplurality of mechanism extensions. The array or strip can be configuredto attached as a unit to the surface, which may facilitate assemblyand/or replacement.

The mechanical extension can extend a distance from the surface to whichit is attached. The distance can define a threshold distance. An objectmore than the threshold distance away from the movement mechanism towhich the mechanical extension is attached will not contact themechanical extension, and the mechanical extension will thus not sensethe object. An impending collision will thus not be detected between theobject and the movement mechanism having the mechanical extensionattached thereto. An object that is at a distance equal to or less thanthe threshold distance will contact the mechanical extension, and themechanical extension will thus sense the object. An impending collisioncan thus be detected between the object and the movement mechanismhaving the mechanical extension attached thereto. Thus, the more themechanical extension extends from the surface to which it is attached,the larger the threshold distance, and thus the more sensitive thesystem 900 will be to impending collisions.

The controller 904 can be configured to determine from a signal from themechanism extension whether a collision is impending. For example, themechanism extension can be configured to transmit a signal to thecontroller 904 in response to the mechanism extension contacting anotherobject. In response to receipt of the signal from the mechanismextension, the controller 904 can determine that a collision isimpending. In some embodiments, the mechanism extension can beconfigured to transmit the signal to the controller 904 only when themechanism extension contacts an object, which may help conserveresources of the mechanism extension and/or the controller 904. In otherembodiments, the mechanism extension can be configured to transmit thesignal to the controller 904 at regular intervals, with the signal beinga binary signal either indicating “yes,” the mechanism extensioncontacted an object since the last transmitted signal, or “no,” themechanism extension has not contacted the object since the lasttransmitted signal. This continuous transmission of signals may allowthe controller 904 to determine whether the movement mechanism has movedaway from the object of potential collision, e.g., is a “no” signal isreceived after a “yes” signal. For another example, the mechanismextension can be configured to have a degree of bend that can becommunicated to the controller 904. Based on the degree of bend of themechanism extension, the controller 904 can determine whether themechanism extension has contacted an object, such as by comparing thereceived degree of bend to a preset default degree of bend of themechanism extension. If the received degree of bend is more than apredetermined amount above or below the preset degree of bend, thecontroller 904 can be configured to determine that the mechanismextension contacted an object and that a collision is imminent. For yetanother example, the mechanism extension can be conductive, can beconfigured to be excited with a low, safe level of voltage, and can beconfigured to detect electrical contact with an object. The controller904 can determine than a collision is impending based on the detectedelectrical contact. For still another example, the controller 904 can beconfigured to determine from signals received from each of a pluralityof mechanism extensions attached to the same movement mechanism adirection of an object with which the movement mechanism may possiblyimminently collide, such as by the varying electrical currents receivedfrom the different ones of the mechanism extensions and/or by thevarying degrees of bend received from the different ones of themechanism extensions. The controller 904 can thus be configured todetermine an imminent collision based on a pattern of mechanismextension signals.

Another type of matter probe includes a cushion attached to a surface ofa movement mechanism or a casing thereof. The cushion can be attached tothe surface in any of a variety of ways, e.g., with adhesive, by beingsprayed onto the surface, etc. In an exemplary embodiment, a pluralityof cushions can be attached to a movement mechanism (directly orindirectly via a casing) and can extend in different directionstherefrom, thereby helping to prevent a collision from occurring in anydirection in which the movement mechanism moves or in any direction thatan object approaches the movement mechanism.

The cushion can have a variety of configurations. The cushion can be,for example, in the form of a rib, a belt similar to a torpedo belt, aballoon, a foam, etc. The cushion as a balloon can be filled with any ofa variety of materials, such as a liquid such as saline, a gas such asair, and a foam such as a reticulated foam. The cushion as a foam can beformed by, for example, molding the foam with materials and processconditions that yield a fused outer surface such that the foam need notbe inside a balloon or have an outer coating. A cushion can have asingle segment or chamber, or a cushion can have a plurality of segmentsor chambers.

The cushion can be pressurized or non-pressurized. A pressurized cushioncan be pressurized in any of a variety of ways. For example, the cushioncan be coupled to a pressure or inflation source provided in themovement mechanism or casing thereof to which the cushion is attached. Anon-pressurized cushion can be non-pressurized in any of a variety ofways. For example, the cushion (e.g., a cushion of reticulated foam) caninclude a small leak (e.g., by including a small hole therein)configured to equalize internal and external pressure and thereby avoidslow pressure-driven dimensional changes that could occur withtemperature and aging while still being responsive to more rapid changesassociated with contact. For another example, a balloon, bladder, orother outer coating of a cushion can be sufficiently porous to allowslow pressure equalization and thereby be sensitive to pressure changesrather than the absolute value of its internal pressure.

The cushion can be configured to deform in response to a force appliedthereto. Contact of the cushion with an object can be determined basedon a pressure applied to the cushion, e.g., by an amount of thecushion's shape deformation. The controller 904 can be configured todetermine from a signal from the cushion whether a collision isimpending, where the signal is indicative of the pressure applied to thecushion. The controller 904 can be configured to determine that thehigher the pressure, the more likely and/or the sooner a collision mayoccur.

The controller 904 of the robotic surgical system 900 can be configuredto trigger performance of a remedial action (also referred to herein asa corrective action) in response to the controller 904 determining thata possible collision may occur based on information sensed by thesensors 908 a, 908N. The remedial action can be configured to helpprevent the collision from occurring. The collision may thus be avoidedentirely, or the collision's adverse effects may be mitigated in theevent that the collision cannot be avoided, such as because one or moreof the colliding objects are moving too quickly to avoid collision afterthe detection of possible collision therebetween. Because the controller904 can be configured to receive data sensed by the sensors 908 a, 908Nin real time with performance of a surgical procedure, the controller904 can be configured to cause the remedial action to be performedduring the surgical procedure and either prevent the collision entirelyor at least reduce adverse effects of the collision should itnevertheless occur.

The remedial action can have a variety of configurations. For example,the remedial action can include any one or more of slowing a movementspeed of the movement mechanism associated with the detected impendingcollision, adjusting the movement the movement mechanism associated withthe detected impending collision in contradiction to the movementthereof requested by a user input to the robotic surgical system 900requesting movement of the surgical instrument coupled to that movementmechanism, moving the object with which the movement mechanismassociated with the detected impending collision may collide, andproviding a notification to a user indicating that the movementmechanism associated with the detected impending collision is subject toan impending collision. The controller 604 can be configured to causethe notification mechanism 902 to provide the notification, e.g., bytransmitting a signal thereto, to the user similar to that discussedabove regarding the notification mechanism 610 providing a cue. Thenotification can include any one or more of an audible signal, a tactilesignal, and a visual signal. The audible, tactile, and visual signalscan be similar to the audible, tactile, and visual signals discussedabove regarding the cue. As mentioned above, in embodiments in which thenotification mechanism 902 is omitted, another portion of the roboticsurgical system 900 can be configured to provide the notification, suchas a display device of the system 900 being configured to display thenotification thereon. In at least some embodiments, the controller 604can be configured to not cause the notification mechanism 902 to providethe if the user's instrument manipulation remains unhindered by theremedial action(s), which may help avoid additional cognitive load onthe user. In such instances, only when no mitigation is possible and adesired movement cannot be fulfilled because of irreducible collisions,the controller 604 can be configured to cause the notification mechanism902 to provide the notification.

FIG. 19 illustrates one embodiment of a process 1000 of the controller904 triggering a remedial action based on data sensed by the sensors 908a, 908N. The controller 904 can be configured to scan 1002 the sensors908 a, 908N for signals. The scanning 1002 can be continuous, or thecontroller 904 can periodically query the sensors 908 a, 908N atregular, predetermined intervals. Continuous scanning 1002 mayfacilitate the identification of potential collisions before they occur.Periodic scanning 1002 may help conserve system resources. The sensors908 a, 908N can all be continuously scanned 1002, can all beperiodically scanned 1002 (at the same or different intervals from oneanother), or at least one of the sensors 908 a, 908N can be continuouslyscanned 1002 and a remainder of the sensors 908 a, 908N can beperiodically scanned 1002.

The scanning 1002 can include the controller 904 sending a query to eachof the sensors 908 a, 908N requesting a reply. Alternatively, thesensors 908 a, 908N can be configured to transmit sensed data to thecontroller 904 without receiving a request for the data from thecontroller 904.

Based on data received from the sensors 908 a, 908N, the controller 904can determine 1004 whether contact is imminent between one of themovement mechanisms 906 a, 906N and another object. The determining 1004can involve any number of different types of analysis, as discussedabove. If contact is determined 1004 as not being imminent, thecontroller 904 can reset 1006 and continue scanning 1002. The resetting1006 can include resetting a clock that counts time for when the nextscanning 1002 should occur and/or can include resetting a display tostop providing notification of an impending collision previouslydetected. In other embodiments, as mentioned above, the notificationmechanism 902 and/or other aspect of the system 900 can instead or alsoprovide the notification instead of the display.

If contact is determined 1004 as being imminent, the controller 904 cancause 1008 the display to provide notification of the detected impendingcollision. The notification on the display can include, for example,text such as “First Arm Contact,” “First Arm Reports Contact,” “FirstArm Impending Contact,” “Second Arm Contact At Joint Three,” “Second ArmReports Medial Contact,” “Contact For First Trocar's Instrument,”“Second Arm Contact From Above,” “Second Arm Contact From Left,” amessage indicating a modification of the instrument pose determined bythe system 900 that would allow the instrument to avoid the collisionand resume uninhibited movement such as a message of “Rotate right toolslightly clockwise or push it left to eliminate external collisions,”etc., and/or an image such as a 3-D graphical indicator (e.g., a coloredwedge, pyramidal frustum, etc.) overlaid on the instrument tipindicating how the instrument's pose can be revised to avoid thecollision with a size or other attribute of the indicator indicating therelative merit (freedom attained) of the revised pose since the user canbe free to adjust one instrument having an overlaid image but notanother instrument having another overlaid image, etc. As mentionedabove, the notification mechanism 902 and/or other aspect of the system900 can instead or also provide the notification instead of the display.The notification includes a visual display in this illustratedembodiment, but as mentioned above, the notification can include any oneor more of visual, tactile, and audible signals.

After determining 1004 that a collision is imminent, the controller 904can cause a clock that counts time to be reset for when the nextscanning 1002 should occur.

In the illustrated embodiment of FIG. 19, the remedial action triggeredby the controller 904 in response to a detected impeding collision,which may be provided before, concurrent with, and/or after thecollision occurs, includes the providing of a notification. FIG. 20illustrates another embodiment of a process 1100 of the controller 904triggering a remedial action based on data sensed by the sensors 908 a,908N, with the remedial action including providing of a notification andcausing an adjustment of the robotic surgical system 900. Thisadjustment may prevent the detected possible collision from occurring atall, even if the user controlling the system 900 does not take anyspecific action in order to prevent the collision.

In the process 1100 of FIG. 20, the controller 904 can be configured toscan 1102 the sensors 908 a, 908N for signals similar to the scanning1002 discussed above. Based on data received from the sensors 908 a,908N, the controller 904 can determine 1104 whether contact is imminentbetween one of the movement mechanisms 906 a, 906N and another object,similar to that discussed above regarding the determining 1004.

If contact is determined 1104 as not being imminent, the controller 904can restore control 1106 and can reset 1108 similar to the resetting1006 discussed above. The restoration 1106 of control can includeallowing movement of the movement mechanisms 906 a, 906N to becontrolled via user input device, which may either be maintainingcontrol via the user input device or to stop automatic control in theevent of a previously detected imminent collision. Additionally oralternatively, the restoration 1106 of control can include returning amovement mechanism that was moved in response to a previously detectedimminent collision back to its pre-detected collision position. Themovement mechanism may have been moved to a mechanically disadvantageousposition and/or may have moved close to another object. Returning themovement mechanism to its previous position may thus allow the movementmechanism to be moved from the mechanically disadvantageous positionand/or move the movement mechanism away from the other object it movedclosed to and may have thus increased its chances of collidingtherewith.

The restoration 1106 of control occurs before the resetting 1108 in thisillustrated embodiment, but the resetting 1108 can occur first, or theresetting 1108 and the restoring 1106 can be concurrent. After therestoration 1106 of control and the resetting 1108, the scanning 1102can continue.

If contact is determined 1104 as being imminent, the controller 904 cancause 1110 automatic movement of a portion of the robotic surgicalsystem 900 to help prevent the impending collision and/or mitigateadverse effects of the collision should it occur. The automatic movementcan include any one or more of at least one of changing a shape of themovement mechanism associated with the sensor associated with thedetected impending collision, changing a shape of the movement mechanismthat may potentially collide with the movement mechanism associated withthe sensor associated with the detected impending collision, slowing aspeed of the movement mechanism associated with the sensor associatedwith the detected impending collision (which may include stopping themovement entirely), adjusting the movement of the movement mechanismassociated with the sensor associated with the detected impendingcollision in contradiction to the movement requested by user input tothe robotic surgical system 900, and moving the other portion of therobotic surgical system 900 that may collide with the movement mechanismassociated with the sensor associated with the detected impendingcollision. Changing a shape of a movement mechanism can include, forexample, bending the movement mechanism, such as bending the movementmechanism at any one or more joints of the movement mechanism.

Also, referring again to FIG. 20, if contact is determined 1104 as beingimminent, the controller 904 can cause 1112 the display to providenotification of the detected impending collision, similar to the setting1008 of the display discussed above. The modification 1110 of controloccurs before the setting 1112 in this illustrated embodiment, but thesetting 1112 can occur first, or the setting 1112 and the modifying 1110can be concurrent. After the modifying 1110 of control and the setting1112, the scanning 1102 can continue.

FIG. 21 illustrates one embodiment of a process 1114 of the controller904, in response to determining 1104 that contact is imminent, causing achange in shape of a movement mechanism in the event that the twoobjects that may imminently collide include two of the movementmechanisms 906 a, 906N. The controller 904 can determine that two of themovement mechanisms 906 a, 906N are the subject of the imminentcollision in any of a variety of ways, such as by receiving signalsindicative of a potential collision from each of the two movementmechanisms' sensors 908 a, 908N simultaneously, thereby indicating thatthe collision is likely with each other. A person skilled in the artwill appreciate that signals may not be received exactly simultaneouslybut nevertheless be considered to be received simultaneously due tofactors such as very small delays in controller 904 processingcapability. For another example, a reflected sound received by amovement mechanism can have a different characteristic pattern ifreflected back from another movement mechanism than from another objectsuch as a person.

If contact is determined 1104 as being imminent, the controller 904 candetermine 1116 if either of the two of the movement mechanisms 906 a,906N at issue are configured to change shape. The determination 1116 caninclude, for example, looking up the two of the movement mechanisms 906a, 906N at issue in a lookup table that indicates a shape changingcapability for each of the movement mechanisms 906 a, 906N in the system900 with a binary true/false value. A movement mechanism can beconfigured to change shape by bending at one or more points along alength thereof, such as if the movement mechanism includes a pluralityof joints or if the movement mechanism includes a flexible elongateshaft.

If neither of the two of the movement mechanisms 906 a, 906N at issueare configured to change shape, the controller 904 can cause 1118automatic movement of a portion of the robotic surgical system 900and/or the provision of a notification, as discussed above. Thenotification can include notice 1120 to the user that a criticalcollision condition exists as a reflection that movement mechanism shapecannot be adjusted.

If only one of the two of the movement mechanisms 906 a, 906N at issueis configured to change shape, the controller 904 can cause that one ofthe movement mechanisms 906 a, 906N to change shape.

If both of the two of the movement mechanisms 906 a, 906N at issue isconfigured to change shape, the controller 904 can change the shape ofboth of the two of the movement mechanisms 906 a, 906N at issue.Alternatively, the controller 904 can be configured to choose 1122 oneof the two of the movement mechanisms 906 a, 906N at issue to change inshape. Changing the shape of only one of the movement mechanisms 906 a,906N at issue may help prevent the movement mechanism whose shape ischanged from risking collision with another object it may move closer toas a result of the shape change. The controller 904 can be configured tochoose 1112 one of the two of the movement mechanisms 906 a, 906N atissue in any of a variety of ways. For example, the controller 904 canbe configured to determine from an electronic model of the roboticsurgical system 900 that reflects a current position of each of themovement mechanisms 906 a, 906N which one of the two movement mechanisms906 a, 906N at issue is less likely to risk collision with anotherobject it may move closer to as a result of the shape change, such as bydetermining which of the two movement mechanisms 906 a, 906N at issue isfarther away from objects adjacent thereto. The controller 904 can cause1122 the chosen one of the two movement mechanisms 906 a, 906N at issueto change in shape.

The controller 904 can update 1124 the model to reflect the caused shapechange and can provide 1126 a notification, as discussed above. Thenotification can include an advisory notice to the user that a shape ofa movement mechanism shape has been adjusted. The updating 1124 occursbefore the setting 1126 in this illustrated embodiment, but the setting1126 can occur first, or the setting 1126 and the updating 1124 can beconcurrent.

As mentioned above, at least some of the methods, systems, and devicesfor controlling movement of a robotic surgical system provided hereincan be configured to facilitate movement of a surgical instrumentbetween different anatomical quadrants. The robotic surgical system cangenerally be configured and used similar to the robotic surgical system200 of FIG. 3. The robotic surgical system can include a computer systemthat includes a display device, a controller, and a plurality ofmovement mechanisms. The robotic surgical system can include at leasttwo movement mechanisms and, optionally, an additional one or moremovement mechanisms for a total of “N” movement mechanisms, where N isgreater than or equal to three. Each of the movement mechanisms can beconfigured to couple to one of a plurality of surgical instruments, asdiscussed herein. The controller can be configured to receive an inputfrom a user requesting movement, relative to a patient, of a one of thesurgical instruments. The user can provide the input using a user inputdevice (e.g., a master tool), as discussed herein. The controller can beconfigured to cause motor(s) of the robotic surgical system to drivemovement of the movement mechanism having the one of the surgicalinstruments coupled thereto, as also discussed herein, thereby causingmovement of the one of the surgical instruments in accordance with theuser's input.

The robotic surgical system can have first and second modes ofoperation. In the first mode of operation, the robotic surgical systemcan be configured to only move the one of the surgical instruments thatthe user requests movement of via the input device. The first mode maytherefore allow the one of the surgical instruments to move relative toall other surgical instruments coupled to the robotic surgical systemvia the other movement mechanisms. In other words, in the first mode,movement of the surgical instrument changes the surgical instrument'sspatial positioning relative to the others of the surgical instruments.In the second mode of operation, the robotic surgical system can beconfigured to move the one of the surgical instruments that the userrequests movement of via the input device in addition to one or more ofthe other surgical instruments coupled to the robotic surgical system.The second mode may therefore allow the two or more of the surgicalinstruments that are moved to maintain a relative position relative toeach other. In other words, in the second mode, the multiple surgicalinstruments that move maintain a fixed spatial relationship to oneanother.

The robotic surgical system can be configured to selectively movebetween the first and second modes in response to a user input to therobotic surgical system, e.g., an input via the user input device suchas a pressing of a “mode” button thereon. The robotic surgical systemcan thus be configured to allow the user to decide when an input to movea surgical instrument should move only that surgical instrument and whenthe surgical instrument should move in conjunction with one or moreother surgical instruments. In at least some embodiments, in addition tothe robotic surgical system being configured to selectively move atleast from the first mode to second mode in response to the user input,the robotic surgical system can be configured to automatically move fromthe second mode to the first mode. The automatic movement can occurafter the robotic surgical system receives the user input requestingmovement of the surgical instrument such that the next user inputrequesting movement of the surgical instrument will move only thatsurgical instrument unless the user manually provides another inputmoving from the first mode to the second mode. The automatic movementfrom the second mode to the first mode may help prevent inadvertentcoordinated movement of surgical instruments, which may be more likelyto occur when user inputs requesting movement of the surgicalinstruments are separated by enough time for the user to possibly forgetthat the robotic surgical system is in the second mode.

When the robotic surgical system is in the first mode, the roboticsurgical system can be configured to move the surgical instrument thatis next instructed by the user to move relative to the others of thesurgical instruments, and when the robotic surgical system is in thesecond mode, the robotic surgical system can be configured to move thesurgical instrument that is next instructed by the user to be moved andto correspondingly move the others of the surgical instruments. Therobotic surgical system can thus be configured in a simple manner inwhich a specific one of the surgical instruments is not pre-selected asa “master” tool that others of the surgical instruments follow as“slave” tools follow, nor are any one or more of the surgicalinstruments pre-selected as “slave” tools since instead all surgicalinstruments move in conjunction with the master tool. Such aconfiguration may be useful when a surgical instrument including acamera is desired to always move in conjunction with another surgicalinstrument to keep the other surgical instrument within the camera'sfield of view. In other embodiments, the robotic surgical system can beconfigured to allow user pre-selection of a one of the surgicalinstruments as the master tool. The user may thus have more flexibilityin deciding how surgical instruments should move since different ones ofthe surgical instruments can be the master tool at different timesduring performance of a surgical procedure. Selection of a surgicalinstrument as the master tool can be accomplished in any of a variety ofways, such as via the user input device or via an IO device coupled tothe robotic surgical system. In addition or alternative to the roboticsurgical system being configured to allow user pre-selection of themaster tool, the robotic surgical system can be configured to allow userpre-selection of one or more of the surgical instruments as slave tools.The user may thus have more flexibility in deciding how surgicalinstruments should move since different ones of the surgical instrumentscan move in conjunction with the master tool at different times duringperformance of a surgical procedure. Selection of surgical instrumentsas slave tools can be accomplished in any of a variety of ways, such asvia the user input device or via an IO device coupled to the roboticsurgical system.

In at least some embodiments, when a camera is one of a plurality ofsurgical instruments in use in a surgical procedure being performedusing the robotic surgical system, the camera can be automaticallyselected by the robotic surgical system as a slave tool. The camera canthus be ensured to move in coordination with other surgicalinstrument(s) being moved, thereby helping to keep the camera trained onthe instrument(s) being moved. No portion of the camera will typicallybe within the camera's field of view, so automatically designating thecamera as a slave tool may facilitate selection the camera as a slavetool in embodiments in which tools selected for coordinated movement areselected based on their visibility on a display.

In at least some embodiments, when a camera is one of a plurality ofsurgical instruments in use in a surgical procedure being performedusing the robotic surgical system, the camera can be selected by a useras either a master tool or a slave tool. The camera's movement may thusbe more in the control of the user. Since no portion of the camera willtypically be within the camera's field of view, in embodiments in whichtools selected for coordinated movement are selected based on theirvisibility on a display, the camera can be selected in another way, suchas by selecting a “camera” icon on the display or by selecting thecamera from a list of surgical instruments currently in use that isprovided on the display. In embodiments in which a list of surgicalinstruments currently in use is provided on a display, whether or notone of the surgical instruments includes a camera, one or moreinstruments can be selected for movement from the list, such as bychecking a check box next to selected one(s) of the instruments or byclicking on text, icon, picture, etc. identifying an instrument.

In at least some embodiments, when a camera is one of a plurality ofsurgical instruments in use in a surgical procedure being performedusing the robotic surgical system, the camera can be configured to becontrolled using a first user input device and the other surgicalinstrument(s) can be configured to be controlled using a second userinput device. The camera can thus be configured to be controlledindependent of the other surgical instrument(s), which may facilitatezooming in/out and/or other field of view changes during movement of anyof the other surgical instrument(s), which may help keep the movinginstrument(s) within the camera's field of view. One example of firstand second input devices includes two hand held devices, one operated bya user's left hand and another operated by the user's right hand.Another example of first and second input devices includes a handhelddevice operated by a user's hand and a foot-operated device operated bythe user's foot.

FIG. 22 illustrates one embodiment of a robotic surgical system 1200configured to facilitate movement of a surgical instrument betweendifferent anatomical quadrants. The robotic surgical system 1200 cangenerally be configured and used similar to the robotic surgical system200 of FIG. 3. For ease of illustration, only some portions of thesystem 1200 are shown in FIG. 22. The robotic surgical system 1200 caninclude a computer system that includes a display device (not shown), acontroller 1202, and a plurality of movement mechanisms (not shown) eachconfigured to removably couple to one of a plurality of surgicalinstruments (not shown). As in this illustrated embodiment, the roboticsurgical system 1200 can include a display space database 1204, a userinput device (UID) database 1206, and a robot geometry database 1208.The databases 1204, 1206, 1208 are shown an individual databases in thisillustrated embodiment, but any one or more of the databases 1204, 1206,1208 can be combined together.

The display space database 1204 can be configured to receive and storecamera view data indicative of a field of view of a camera coupled tothe robotic surgical system 1200 via a movement mechanism thereof,patient data indicative of data related to a patient (e.g., anatomicaldata, etc.) to have or having a surgical procedure performed thereonusing the robotic surgical system 1200, and simulation data indicativeof a simulated surgical procedure performed on the patient beforecommencement of an actual surgical procedure on the patient.

The UID database 1206 can be configured to receive and store input fromeach user input device coupled to the robotic surgical system 1200, suchas poses of user input devices configured to be operated by hand as wellas any other signals such as the state of various switches they mayprovide, the values of signals provided by foot-operated user inputdevices, and can be configured to receive and store field of view (FOV)data and direction of view (DOV) data from the display space database1204. The UID database 1206 can be configured to provide command data tothe controller 1202 indicative of input the robotic surgical system 1200receives from a user input device so the controller 1202 can actuate theuser input device signal. In other words, so the controller 1202 cancause appropriate action to be taken in response thereof, e.g., movebetween the first and second modes, cause movement of one or moresurgical instruments in accordance with in the user input. The UIDdatabase 1206 can be configured to provide cursor data to anintersection evaluator 1210 configured to facilitate association of acursor that may be overlaid on an image with objects in the imageitself. The intersection evaluator 1210 is shown as a separate elementfrom the controller 1202 in this illustrated embodiment, but thecontroller 1202 can perform the functions of the intersection evaluator1210.

The robot geometry database 1208 can be configured to receive and storejoint angles of the movement mechanisms, positions of a base or frame ofthe robotic surgical system 1202, lengths of the movement mechanisms,lengths of the surgical instruments coupled to the movement mechanisms,and data regarding articulation of the surgical instruments coupled tothe movement mechanisms. The robot geometry database 1208 can beconfigured to provide data stored therein to the intersection evaluator1210 to facilitate association of cursors with objects such as any oneor more of instruments, patient anatomy, and derived entities. The robotgeometry database 1208 can be configured to provide data stored thereinto the controller 1202 to facilitate various processing operations ofthe controller 1202. The robot geometry database 1208 can be configuredto receive cursor data from the intersection evaluator 1210, e.g., fromthe controller 1202, thereby allowing non-geometric or “meta” data to beincluded in the data reflective of the robot surgical system'sconfiguration. The cursor data can include identification of which oneof the surgical instruments coupled to the movement mechanisms is themaster tool and which one or more of the surgical instruments coupled tothe movement mechanisms are the slave tools that follow the movement ofthe master tool when the robotic surgical system 1200 is in the secondmode.

FIGS. 23 and 24 illustrate one embodiment of a process of a roboticsurgical system facilitating movement of a surgical instrument betweendifferent anatomical quadrants. A first portion 1300 of the process inFIG. 23 is generally a “gathering” process in which one or more surgicalinstruments coupled to the robotic surgical system (e.g., removablycoupled to one or more movement mechanisms of the robotic surgicalsystem) are selected as tools to move in conjunction with one another tomaintain fixed spatial positioning therebetween, e.g., the slave tool(s)are gathered. A second portion 1302 of the process in FIG. 24 isgenerally a “follow” process in which the selected one or moreinstruments are moved in accordance with user-instructed movement.

A user of the robotic surgical system can indicate 1304 that thegathering process 1300 should begin. The user can provide the indication1304 in any of a variety of ways, as discussed above. In response to theindication 1304, the robotic surgical system (e.g., a controllerthereof) can store 1306 a present configuration of the robotic surgicalsystem, e.g., store a present configuration of the movement mechanismsthereof. The stored 1306 present configuration may be used by the systemin the event that the user would like to restore positioning of thesurgical instruments coupled to the system to their position prior tothe movement of the selected tool(s) that occurs later in the followprocess 1302.

The robotic surgical system (e.g., the controller thereof) can suggest1308 one of the surgical instruments to the user as a first tool formovement. The suggestion 1308 can, as in this illustrated embodiment, bebased on where a user-controlled cursor is currently positioned on adisplay screen coupled to the robotic surgical system. A one of thesurgical instruments closest to the cursor can be the suggested 1308 oneof the instruments. The robotic surgical system (e.g., the controllerthereof) can cause a marker (e.g., an “X,” a dot, a bullseye, aquatrefoil, a square, etc.) to be visible on the screen that identifiesthe suggested 1308 one of the instruments. The user can provide an inputto the system (e.g., via a user input device) indicating 1310 whetherthe suggested 1308 one of the instruments is an instrument that the userwould like to move. If not, the robotic surgical system (e.g., thecontroller thereof) can suggest 1312 another one of the surgicalinstruments. As in this illustrated embodiment, the next suggested 1312one of the surgical instruments can be a one of the surgical instrumentsthat is next nearest to the cursor's current position. Once thesuggested 1308 instrument is an instrument that the user would like tomove, the robotic surgical system (e.g., the controller thereof) can add1314 the selected instrument to a “move group” that includes allinstruments selected as tools to move. The robotic surgical system(e.g., the controller thereof) can cause a marker (e.g., an “X,” a dot,a bullseye, a quatrefoil, a square, etc.) to be visible on the screenthat identifies the added 1314 one of the instruments. The marker usedto identify a suggested 1308 instrument can be different than the markerused to identify an added 1314 instrument, which may help the usereasily visually identify selected tools.

A location of where the marker is overlaid on the image of the selectedinstrument can be determined in any of a variety of ways. The cameraimage on the display has a plurality of pixels that each corresponds toa particular direction of incoming light. When the cursor is placedwithin the camera's field of view on the display, the user is specifyinga direction in the field of view, which generally corresponds to adirection with respect to an elongate shaft of the camera. The roboticsurgical system (e.g., the controller thereof) can compute a first linefrom the camera's lens in the direction of the field of view and asecond line along a longitudinal axis of the selected instrument'selongate shaft (which can be known due to the known geometry of themovement mechanism to which the selected instrument is coupled). Anarticulated end effector at a distal end of the selected instrument'selongate shaft can, in at least some embodiments, be accounted for byadding a bend to a distal end of the second line. The first and secondlines can be in three dimensions. The robotic surgical system (e.g., thecontroller thereof) can find where the first line comes closest to thesecond line and place the cursor at that location. The marker can thusnot only mark the selected instrument but a location on the selectedinstrument. The robotic surgical system can allow the user to adjust aposition of the marker on the selected instrument, such as by draggingand dropping of the marker.

The first added 1314 instrument can serve as a master tool. The user canindicate 1316 whether or not the user would like to select another oneof the instruments, such as by moving 1318 the cursor toward another oneof the surgical instruments. If no further tools are selected, theselected one tool can move on its own, e.g., with the robotic surgicalsystem operating in the first mode. If one or more additional tools areselected and added 1314 to the group, the selected tools can all move inaccordance with the user-instructed movement of the master tool, e.g.,with the robotic surgical system operating in the second mode.

Once the user has selected all of the tools, the user can choose 1320 tocontinue to the follow process 1302, e.g., choose to move the selectedtool(s), or to cancel 1322 the tool selections. The cancellation 1322can allow the user to either start the tool selection process again orcancel the second mode and perform some other aspect of the surgicalprocedure. The user's choice 1320 can be indicated in any of a varietyof ways, such as via the user input device. Examples of cancelling 1322include the user moving the cursor on the display screen to select a“cancel” icon thereon. In this illustrated embodiment, the user choosing1320 to proceed with movement of the selected tool(s) is indicated bythe user providing 1324 an input to the robotic surgical systemindicating desired movement of the master tool (e.g., providing an inputto the user input device). The robotic surgical system (e.g., thecontroller thereof) can cause 1326 movement of the master tool and anyselected slave tools in accordance with the provided 1324 input suchthat the selected slave tool(s) move in correspondence to the mastertool's movement. To facilitate this coordinated movement, the roboticsurgical system (e.g., the controller thereof) can map the master tooland the selected slave tool(s) to an electronic constellation defined bythe coordinate systems of each of the master tool and the selected slavetool(s) and including points representative of the tools' locations. Theelectronic constellation can correspond to the configuration of therobotic surgical system mentioned above. The user can provide 1328additional movement instructions to cause movement of the selectedtool(s). After all movement of the selected instrument(s) desired by theuser has been accomplished, the robotic surgical system (e.g., thecontroller thereof) can end 1330 the process and allow other aspects ofthe surgical procedure to be performed, which may include one or moreadditional gather/follow processes.

FIG. 25 illustrates one embodiment of a display that can be shown on adisplay screen of a robotic surgical system to facilitate movement of asurgical instrument between different anatomical quadrants. In thisillustrated embodiment, a field of view 1400 of a camera (not shown)visualizing a patient has therein an omentum 1402 of the patient, aspleen 1404 of the patient, and an area 1406 including a smallintestine, large intestine, vascular cascade, and arches of the patient.Three surgical instruments, a first grasper 1408, a second grasper 1410,and a suction/irrigation probe 1412, have been positioned relative tothe patient and are also in the field of view 1400. The display includesa marker 1414 thereon indicating that the robotic surgical system is ina “gather” mode, e.g., is at a start of the gathering process 1300 ofFIG. 23. The marker 1414 includes emphasized (glowing) text in thisillustrated embodiment, but the marker 1414 can have any of a variety ofconfigurations, e.g., different text, an icon, etc. The display in thisillustrated embodiment also has a cursor 1416 and a first toolidentifying marker 1418 thereon. The cursor 1416 is an “X” in thisillustrated embodiment, but the cursor 1416 can have any of a variety ofconfigurations, e.g., a dot, an hourglass, an arrow, etc. The first toolidentifying marker 1418 in this illustrated embodiment is a CG (centerof gravity) symbol, but the cursor 1416 can have any of a variety ofconfigurations, e.g., a solid dot, a triangle in outline, an “S,” anarrow, etc. The cursor 1416 and the first tool identifying marker 1418can be used as discussed above regarding the gathering process 1302 ofFIG. 23. In FIG. 25, the slave tool identifying marker 1418 has beenpositioned on the display at the one of the surgical instruments 1408,1410, 1412 closest to the current position of the cursor 1416 toindicate a suggested one of the surgical instruments 1408, 1410, 1412 asa tool for movement.

FIG. 26 shows the display of FIG. 25 at a subsequent point in time andin a zoomed-in view. FIG. 16 also shows a wrist mechanism 1408 a, 1410 aof the first and second graspers 1408, 1410. In this illustratedembodiment, the first tool identifying marker 1418 remains on the firstgrasper 1408, thereby indicating that the first grasper 1408 has beenselected by the user as a tool for movement, and a second toolidentifying marker 1420 is on the second grasper 1410, therebyindicating that the user also selected the second grasper 1410 as a toolfor movement. The second tool identifying marker 1420 is CG symbol inthis illustrated embodiment, but as mentioned above, tool identifyingmarkers can have other configurations. The display in this example alsohas thereon a central marker 1422 positioned on the display at a centerpoint of the markers 1418, 1420 of all the selected tools 1408, 1410.For two selected instruments, the central marker 1422 can be at amidpoint between the two markers 1418, 1420. For three or more selectedinstruments, the central marker can be at a geometric centroid of themarkers of selected instruments.

The central marker 1422 represents a reference point of movement for allthe selected tools 1408, 1410 during movement of the selected tools1408, 1410. The location of the markers 1418, 1420 on their respectivetools 1408, 1410 thus has meaning for the upcoming coordinated movementof the selected tools 1408, 1410. The central marker 1422 may help theuser visualize possible coordinated movement before the movement occurs.Because the robotic surgical system can know the location andorientation of each of the surgical instruments in use due to knowingthe geometry of the movement mechanisms to which the surgicalinstruments are coupled, the robotic surgical system can map theselected ones of the surgical instruments 1408, 1410 to a common(shared) coordinate system. The central marker 1422 can be placed basedon the common coordinate system.

In at least some embodiments, instead of using the calculated referencepoint represented on the display by the central marker 1422, the roboticsurgical system can be configured to allow the user to select thereference point, such as by moving the cursor to a location on thedisplay and clicking at that location.

FIG. 27 illustrates another embodiment of a display that can be shown ona display screen of a robotic surgical system to facilitate movement ofa surgical instrument between different anatomical quadrants. In thisillustrated embodiment, a field of view 1500 of a camera (not shown)visualizing a patient has therein an omentum 1502 of the patient and anintestine 1504 of the patient. Two surgical instruments, a first grasper1506 and a second grasper 1508, have been positioned relative to thepatient and are also in the field of view 1500. First and second trocars1510, 1512 through which the first and second graspers 1506, 1508 haverespectively been advanced into the patient are also in the field ofview 1500.

As in this illustrated embodiment, the display includes a legend 1514thereon indicating select, gather, and follow modes in which the roboticsurgical system can operate. The legend 1514 includes a list of text inthis illustrated embodiment, but the legend 1514 can have otherconfigurations, e.g., include other text, include icons, etc. The legend1514 can appear at any location on the display. The select modegenerally corresponds to a beginning portion of the gather process 1300of FIG. 23, the gather mode generally corresponds to a remaining portionof the gather process 1300 of FIG. 23, and the follow mode generallycorresponds to the follow process 1302 of FIG. 24. None of the select,gather, and follow modes are currently active, as indicated by the modesall being shown in shaded text. With the select, gather, and followmodes all being inactive, the display may be used by a user in any of anumber of ways, such as for visualization of surgical space, foradjustment of the field of view 1500, etc.

FIG. 28 shows the display of FIG. 27 at a subsequent point in time.Select mode is now active, as indicated by the text “SELECT” beingbolded in the legend 1514. Select mode generally reflects a mode foridentifying of instruments to participate in a coordinated move. In atleast some embodiments, only a selected one of the select, follow, andgather modes may appear in the legend 1514, e.g., the text “SELECT” mayremain while the text “GATHER” and “FOLLOW” disappears. Select modebeing activated generally corresponds to the indication 1304 of FIG. 23and can be accomplished in any of a variety of ways, such as by the userpositioning a cursor 1516 on the text “SELECT” and clicking thereon.FIG. 28 shows the cursor 1516 thereon and a suggested one of theinstruments 1506, 1508 that is closest to the cursor 1516 marked with afirst marker 1518. This suggestion generally corresponds to thesuggesting 1308 of FIG. 23.

FIG. 29 shows the display of FIG. 28 at a subsequent point in time.Select mode is still active, as indicated by the text “SELECT” stillbeing bolded in the legend 1514. The first grasper 1506 has beenselected as an instrument to move (e.g., indicated by the user similarto the indicating 1310 of FIG. 23), as indicated by the first marker1518 still being displayed thereon. The user has moved the cursor 1516to an unselected instrument, the second grasper 1508, which is suggestedas an instrument to move by being marked with a second marker 1520. Thissuggestion generally corresponds to an iteration of the suggesting 1308of FIG. 23. As in this illustrated embodiment, a connecting line 1522can be shown on the display that connects the markers 1518, 1520. Theconnecting line 1522 may provide the user with a visual frame ofreference. The connecting line 1522 is dashed on the display because thesecond grasper 1508 has not yet been indicated by the user as being atool to move.

FIG. 30 shows the display of FIG. 29 at a subsequent point in time.Gather mode is now active, as indicated by the text “GATHER” beingbolded in the legend 1514. Gather mode generally reflects a mode for theuser to identify what subsequent motion commands (e.g. by moving a userinput device) should mean to the selected instruments. This identifyingcan include identifying one of the instruments as a master, with theothers being slaves to follow the master according to some prescribedbehavior such as “maintain the spatial relationship you now have,”“maintain the relationship you have as seen in the present image,” etc.The identifying can include identifying the centroid of the selectedinstruments as the master, or the center of the field of view. The usercan choose the gather mode in any of a variety of ways, such as bypositioning the cursor 1516 on the text “GATHER” and clicking thereon.In at least some embodiments, the robotic surgical system canautomatically move from the select mode to the gather mode in responseto the user indicating that no more tools are to be selected formovement, similar to the indicating 1316 of FIG. 23. The display in thegather mode has thereon a central marker 1524 positioned on the displayat a center point of the markers 1518, 1520 of all the selected tools1506, 1508, similar to the central marker 1422 of FIG. 26. Theconnecting line 1522 is solid on the display because the second grasper1508 has been indicated by the user as being a tool to move such thatthere have been at least two instruments selected so that a centralpoint can be calculated by the robotic surgical system (e.g., by thecontroller thereof). The cursor 1516 can be moved by the user indicatingthat the central marker 1524 is the element that will be controlled inthe subsequent follow mode.

FIG. 31 shows the display of FIG. 30 at a subsequent point in time.Follow mode is now active, as indicated by the text “FOLLOW” beingbolded in the legend 1514. The user can choose the follow mode in any ofa variety of ways, such as by positioning the cursor 1516 on the text“FOLLOW” and clicking thereon. In at least some embodiments, the roboticsurgical system can automatically move from the gather mode to thefollow mode in response to the user indicating that no more tools are tobe selected for movement, similar to the choosing 1320 of FIG. 23. Thecursor 1516 has a different appearance in the gather mode than itsappearance in the select and follow modes, which may help signal to theuser that the selected instruments 1506, 1508 are ready to be moved. Thecursor's form may remind the user that both translational and rotationalmovements of the selected instruments will be made in response to usercommands. Such movements may be advantageous in moving an organ to avoidits distortion or excess tension on its attachments. The central marker1524 can serve as a tether that the user can manipulate to move theselected tools 1506, 1508, as indicated by the cursor 1516 beingpositioned at the central marker 1524. The selected tools 1506, 1508 inFIG. 31 have not yet been moved from their position in FIGS. 27-30.

FIG. 32 shows the display of FIG. 32 at a subsequent point in time.Follow mode is still active, as indicated by the text “FOLLOW” stillbeing bolded in the legend 1514. The user has provided input to thesystem (e.g., via a user input device similar to the providing 1324 ofFIG. 24), and the system in response to the input has moved the firstand second graspers 1506, 1508 in coordination with each other, similarto the causing 1326 of FIG. 24. The first and second markers 1518, 1520have traveled with their respective first and second graspers 1506,1508, as has the connecting line 1522 and the central marker 1524. Thetraveling of the cursor 1516, as moved by the user, may help the userprovide input to the system (e.g., via the user device) that moves theselected instruments 1506, 1508 in a way desired by the user since allgraphical overlays (markers, lines, text, etc.) should still becongruent with the instruments if the robot surgical system has movedcorrectly without collisions or faults occurring. The movement of thefirst and second graspers 1506, 1508 is shown in real time on thedisplay in this illustrated embodiment, and the first, second, andcentral markers 1518, 1520, 1524, the connecting line 1522, and thecursor 1516 can move on the display in real time with the movement ofthe selected instruments 1506, 1508, which may help the user move theselected instruments 1506, 1508 in a desired way and make any neededmovement corrections in real time with the selected instruments'movement. After all movement of the selected instruments 1506, 1508desired by the user has been accomplished, the user can deactivate thefollow mode, such as by clicking on either the “GATHER” text or the“SELECT” text in the legend 1514. The deactivation can be similar to theending 1330 of FIG. 24.

One example of movement using the reference point defined by the centralmarker 1524, whether its location is calculated by the system orselected by the user, is the user requesting rotation about a z axis,e.g., abduction. In response, the connecting line 1522 can rotate abouta vertical axis extending through the central marker 1524 such that oneof the instruments 1506, 1508 extends and the other of the instruments1506, 1508 retracts while maintaining a same distance between theinstruments 1506, 1508. The selected instruments can move in response tocursor 1516 movements, not the tool markers or connecting line 1522 inresponse to center marker 1524. The cursor 1516 that has moved hasbreadth, depth, and volume, so that rotating the cursor 1516 has visualsignificance to the user since it is virtually connected to theappropriate user input device. In contrast, rotating a point on a lineis not visually apparent. In at least some embodiments, graphicalobjects on the display can be eliminated, once the select and gatherphases are complete since they may no longer be needed. For anotherexample of movement using the reference point defined by the centralmarker 1524, whether its location is calculated by the system orselected by the user, is the user requesting rotation about a y axis,e.g., supination. In response, the tool markers 1518, 1520 would move ina same direction along a circumference circle whose diameter is theconnecting line 1522. For yet another example of movement using thereference point defined by the central marker 1524, whether its locationis calculated by the system or selected by the user, is a cameratracking the central marker 1524 during movement of the selectedinstrument(s).

FIG. 33 illustrates yet another embodiment of a display that can beshown on a display screen of a robotic surgical system to facilitatemovement of a surgical instrument between different anatomicalquadrants. In this illustrated embodiment, a field of view of a camera(not shown) visualizing a patient has therein an omentum 1600 of thepatient, a small intestine 1602 of the patient, a descending colon 1604of the patient, an ascending colon 1606 of the patient, and a stomach1608 of the patient. Two surgical instruments, a first grasper 1610 anda second grasper 1612, have been positioned relative to the patient andare also in the field of view.

FIG. 34 shows the display of FIG. 33 at a subsequent point in time. FIG.34 shows a cursor 1614 on the display that the user has moved to thesecond grasper 1612. The cursor 1614 can, as mentioned herein, haveother configurations, such as like the hand cursor 1516 of FIG. 31 thatcan depict to the user what user input device hand motions will achieve.FIG. 35 shows the display of FIG. 34 at a subsequent point in time. Theuser has selected the first and second graspers 1610, 1612 asinstruments to move, as indicated by a solid connecting line 1616extending therebetween and a central marker 1618 being on the line 1616.The selected instruments 1610, 1612 do not have markers thereonindicating that they have been selected, which may help reduce clutteron the display. The selected tools 1610, 1612 in FIG. 35 have not yetbeen moved from their position in FIGS. 33 and 34. FIG. 36 shows thedisplay of FIG. 35 at a subsequent point in time. The user has providedinput to the system (e.g., via a user input device similar to theproviding 1324 of FIG. 24), and the system in response to the input hasmoved the first and second graspers 1610, 1612 in coordination with eachother, similar to the causing 1326 of FIG. 24.

The devices disclosed herein can also be designed to be disposed ofafter a single use, or they can be designed to be used multiple times.In either case, however, the device can be reconditioned for reuse afterat least one use. Reconditioning can include any combination of thesteps of disassembly of the device, followed by cleaning or replacementof particular pieces and subsequent reassembly. In particular, thedevice can be disassembled, and any number of the particular pieces orparts of the device can be selectively replaced or removed in anycombination. Upon cleaning and/or replacement of particular parts, thedevice can be reassembled for subsequent use either at a reconditioningfacility, or by a surgical team immediately prior to a surgicalprocedure. Those skilled in the art will appreciate that reconditioningof a device can utilize a variety of techniques for disassembly,cleaning/replacement, and reassembly. Use of such techniques, and theresulting reconditioned device, are all within the scope of the presentapplication.

Preferably, components of the invention described herein will beprocessed before use. First, a new or used instrument is obtained and ifnecessary cleaned. The instrument can then be sterilized. In onesterilization technique, the instrument is placed in a closed and sealedcontainer, such as a plastic or TYVEK bag. The container and instrumentare then placed in a field of radiation that can penetrate thecontainer, such as gamma radiation, x-rays, or high-energy electrons.The radiation kills bacteria on the instrument and in the container. Thesterilized instrument can then be stored in the sterile container. Thesealed container keeps the instrument sterile until it is opened in themedical facility.

Typically, the device is sterilized. This can be done by any number ofways known to those skilled in the art including beta or gammaradiation, ethylene oxide, steam, and a liquid bath (e.g., cold soak).An exemplary embodiment of sterilizing a device including internalcircuitry is described in more detail in U.S. Pat. Pub. No. 2009/0202387filed Feb. 8, 2008 and entitled “System And Method Of Sterilizing AnImplantable Medical Device.” It is preferred that device, if implanted,is hermetically sealed. This can be done by any number of ways known tothose skilled in the art.

One skilled in the art will appreciate further features and advantagesof the invention based on the above-described embodiments. Accordingly,the invention is not to be limited by what has been particularly shownand described, except as indicated by the appended claims. Allpublications and references cited herein are expressly incorporatedherein by reference in their entirety.

What is claimed is:
 1. A surgical system, comprising: anelectromechanical arm configured to removably couple to a surgicalinstrument, the electromechanical arm being configured to move so as tomove the surgical instrument removably coupled thereto relative to apatient on which a surgical procedure is being performed; a camerahaving a field of view; and a controller configured to receive aninstruction from a user requesting movement of the surgical instrumentfrom a start location outside of the field of view, move the surgicalinstrument in accordance with the instruction, provide a constant cueduring the movement of the surgical instrument, the cue changing duringthe movement of the surgical instrument in relation to a location of thesurgical instrument relative to the field of view, and stop the cue whenthe surgical instrument moves into the field of view.
 2. The system ofclaim 1, wherein the cue is at least one of a visual cue and an auditorycue.
 3. The system of claim 1, wherein the controller is configured torepeatedly determine a current bearing of a longitudinal axis of thesurgical instrument as the surgical instrument is moved by theelectromechanical arm, and the controller is configured to change thecue during the movement of the surgical instrument based on anintersection of the current bearing with the field of view.
 4. Thesystem of claim 1, wherein, after the cue has been stopped, thecontroller is configured to receive a second instruction from the userindicating that the user desires to adjust a position the surgicalinstrument, and, if the surgical instrument moves out of the field ofview as a result of the adjusting, provide a notification to the userindicating that the surgical instrument has moved out of the field ofview.
 5. The system of claim 1, further comprising a display deviceconfigured to display an image of the field of view visualized by thecamera, wherein the controller is configured to repeatedly update theimage throughout the movement of the surgical instrument.
 6. A surgicalmethod, comprising: displaying an image on a display device showing afield of view of a camera visualizing a surgical area within a patient;providing a constant cue to a user positioning a distal working end of asurgical instrument relative to the patient using a robotic surgicalsystem to which the surgical instrument is removably coupled, the distalworking end of a surgical instrument starting its movement from outsidethe field of view, the cue changing over time in relation to a locationof the distal working end of the surgical instrument relative to thefield of view displayed on the display device; and stopping provision ofthe cue in response to the distal working end of the surgical instrumententering the field of view displayed on the display device.
 7. Themethod of claim 6, wherein the cue is at least one of a visual cue andan auditory cue.
 8. The method of claim 6, wherein the cue includes alight that changes over time in at least one of brightness and color inrelation to the location of the distal working end of the surgicalinstrument relative to the field of view displayed on the displaydevice.
 9. The method of claim 6, wherein the cue includes an icon shownon the display device that changes over time in relation to the locationof the distal working end of the surgical instrument relative to thefield of view displayed on the display device.
 10. The method of claim6, wherein the cue includes an audible sound that changes over time inrelation to the location of the distal working end of the surgicalinstrument relative to the field of view displayed on the displaydevice.
 11. The method of claim 10, wherein the audible sound changingincludes changing a pitch of the sound or changing an amplitude of thesound.
 12. The method of claim 6, wherein the cue changing over time inrelation to the location of the distal working end of the surgicalinstrument relative to the field of view displayed on the display deviceis based on an intersection of the field of view and a trajectorydefined by a longitudinal axis of an elongate shaft of the surgicalinstrument.
 13. The method of claim 6, wherein the cue changing overtime in relation to the location of the distal working end of thesurgical instrument relative to the field of view displayed on thedisplay device is based on an intersection of the field of view and atrajectory defined by a longitudinal axis of a trocar through which thesurgical instrument is advanced into the patient.
 14. The method ofclaim 6, wherein the cue changes over time based on at least one of adirection that the distal working end of the surgical instrument ismoving in relative to the field of view displayed on the display device,a distance between the distal working end of the surgical instrument andthe field of view displayed on the display device, and whether thedistal working end of the surgical instrument is moving closer to orfarther from the field of view displayed on the display device.
 15. Themethod of claim 6, further comprising, after the cue has stopped beingprovided, adjusting a position of the distal working end of the surgicalinstrument, and, if the distal working end of the surgical instrumentmoves out of the field of view as a result of the adjusting, providing anotification to the user indicating that the distal working end of thesurgical instrument has moved out of the field of view.
 16. The methodof claim 6, wherein the robotic surgical system includes a plurality ofelectromechanical arms, the surgical instrument being removably coupledto one of the arms, the location of the distal working end of thesurgical instrument relative to the field of view displayed on thedisplay device being determined based on a position of the one of thearms to which the surgical instrument is removably coupled.
 17. Asurgical method, comprising: receiving an input from a user indicatingthat the user desires to move a surgical instrument currently out of afield of view of a camera visualizing a surgical area of a patient, thesurgical instrument being coupled to an electromechanical arm of arobotic surgical system that moves the surgical instrument relative tothe patient; calculating a bearing of a trajectory line between atrocar, through which the surgical instrument is advanced, and a pointin the surgical area being visualized in the field of view; repeatedlydetermining a current bearing of a longitudinal axis of the surgicalinstrument as the surgical instrument is moved by movement of theelectromechanical arm; providing a cue indicative of a differencebetween the bearing of the trajectory line and the current bearing ofthe longitudinal axis, the cue changing over time based on an amount ofthe difference as the surgical instrument moves; and stopping provisionof the cue when the difference becomes substantially zero.
 18. Themethod of claim 17, wherein providing the cue includes at least one ofproviding a visual cue and providing an auditory cue.
 19. The method ofclaim 17, further comprising displaying an image on a display deviceshowing the field of view of the camera.
 20. The method of claim 17,further comprising, after the cue has stopped being provided, receivinga second input from the user indicating that the user desires to adjusta position the surgical instrument currently in the field of view of thecamera, and, if the surgical instrument moves out of the field of viewas a result of the adjusting, providing a notification to the userindicating that the surgical instrument has moved out of the field ofview.